[{"data":1,"prerenderedAt":5939},["ShallowReactive",2],{"blog-\u002Fblog\u002Ffrom-safetensors-to-tokens-executing-a-transformer-step-by-step\u002F":3,"blog-translation-\u002Fblog\u002Ffrom-safetensors-to-tokens-executing-a-transformer-step-by-step\u002F":5908,"blog-series-\u002Fblog\u002Ffrom-safetensors-to-tokens-executing-a-transformer-step-by-step\u002F":5928,"i-simple-icons:github":5929,"i-simple-icons:linkedin":5933,"i-simple-icons:bluesky":5935,"i-simple-icons:x":5937},{"id":4,"title":5,"author":6,"body":7,"cover":5900,"date":5901,"description":5902,"extension":5903,"featured":5904,"kind":5905,"language":5906,"meta":5907,"modified":5908,"navigation":933,"path":5909,"published":933,"readingTime":5910,"seo":5911,"seriesKey":5908,"seriesOrder":5908,"seriesTitle":5908,"sitemap":5912,"stem":5922,"tags":5923,"translationKey":5908,"__hash__":5927},"blog\u002Fblog\u002Ffrom-safetensors-to-tokens-executing-a-transformer-step-by-step.md","From safetensors to tokens, executing a Transformer step by step","Lucas Jahier",{"type":8,"value":9,"toc":5888},"minimark",[10,22,25,34,42,47,50,486,847,854,857,860,864,880,883,897,1095,1098,1104,1107,1206,1278,1282,1285,1725,2073,2079,2082,2251,2254,2445,2449,2452,2802,2805,2811,2821,2828,2945,2948,2951,3009,3013,3016,3505,3508,3511,3651,3654,3736,3804,4394,4449,4682,4685,4691,4698,4701,4704,4844,4847,4853,4939,4942,4946,4949,5069,5072,5175,5179,5182,5226,5229,5336,5339,5345,5348,5496,5500,5507,5510,5516,5526,5777,5783,5790,5794,5797,5803,5806,5809,5812,5817,5821,5828,5832,5885],[11,12,13,14,21],"p",{},"We asked ",[15,16,20],"a",{"href":17,"rel":18},"https:\u002F\u002Fhuggingface.co\u002Fmeta-llama\u002FLlama-3.2-3B-Instruct",[19],"nofollow","Llama 3.2 3B Instruct"," a short question. \"What is the capital of France? Answer in one word.\"",[11,23,24],{},"It returned Paris, a full stop and an end-of-turn token. Three token IDs for a one-word answer. We will follow the operations that produced them, from the first embedding lookup to the last token choice.",[11,26,27,28,33],{},"The Rust extracts come from ",[15,29,32],{"href":30,"rel":31},"https:\u002F\u002Fgithub.com\u002Fstratorys\u002Ffoundry",[19],"Foundry",", our inference engine. We captured the reference values with Transformers and checked Foundry against the same input IDs. The weights use BF16, a format that stores each number in 16 bits. The run data and comparison report are included at the end of the article.",[11,35,36,37,41],{},"The previous article covers ",[15,38,40],{"href":39},"\u002Fblog\u002Finside-llm-model-files-understanding-safetensors-and-gguf\u002F","how the weights are stored on disk",". Here we look at how the runtime uses them.",[43,44,46],"h2",{"id":45},"from-a-prompt-to-a-token","From a prompt to a token",[11,48,49],{},"The model gives each possible next token a score based on the tokens already in the context. The runtime picks one, adds it to the sequence and calls the model again. Softmax can turn the scores into a probability for each token.",[51,52,64,69,131,135,137],"svg",{"style":53,"className":54,"viewBox":56,"width":57,"height":58,"dataWidth":59,"dataHeight":60,"xmlns":61,"xmlnsXLink":62,"xmlns:h5":63},"vertical-align: -NaNem;; display: block; margin: 0 auto;",[55],"typst-doc","0 0 97.000 14.000","8.818181818181818em","1.2727272727272727em","97.000","14.000","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","http:\u002F\u002Fwww.w3.org\u002F1999\u002Fxlink","http:\u002F\u002Fwww.w3.org\u002F1999\u002Fxhtml",[65,66,68],"style",{"type":67},"text\u002Fcss",".typst-text {\n  pointer-events: bounding-box;\n}\n\n.tsel span,\n.tsel {\n  left: 0;\n  position: fixed;\n  text-align: justify;\n  white-space: nowrap;\n  width: 100%;\n  height: 100%;\n  text-align-last: justify;\n  color: transparent;\n  white-space: pre;\n}\n.tsel span::-moz-selection,\n.tsel::-moz-selection {\n  color: transparent;\n  background: #7db9dea0;\n}\n.tsel span::selection,\n.tsel::selection {\n  color: transparent;\n  background: #7db9dea0;\n}\n.pseudo-link {\n  fill: transparent;\n  cursor: pointer;\n  pointer-events: all;\n}\nsvg {\n  fill: none;\n}\n\n.outline_glyph path,\npath.outline_glyph {\n  fill: var(--glyph_fill);\n  stroke: var(--glyph_stroke);\n}\n\n.outline_glyph path,\npath.outline_glyph {\n  transition: 0.2s fill stroke;\n}\n.hover .typst-text {\n  --glyph_fill: #66bab7;\n  --glyph_stroke: #66bab7;\n}\n\n.typst-jump-ripple,\n.typst-debug-react-ripple {\n  width: 0;\n  height: 0;\n  background-color: transparent;\n  position: absolute;\n  border-radius: 50%;\n}\n.typst-jump-ripple {\n  border: 1px solid #66bab7;\n}\n.typst-debug-react-ripple {\n  border: 1px solid #cb1b45;\n}\n@keyframes typst-jump-ripple-effect {\n  to {\n    width: 10vw;\n    height: 10vw;\n    opacity: 0.01;\n    margin: -5vw;\n  }\n}\n@keyframes typst-debug-react-ripple-effect {\n  to {\n    width: 3vw;\n    height: 3vw;\n    opacity: 0.01;\n    margin: -1.5vw;\n  }\n}\n",[70,71,74,81,86,91,96,101,106,111,116,121,126],"defs",{"className":72},[73],"glyph",[75,76],"path",{"id":77,"className":78,"d":80},"gl3Hi1Q0",[79],"outline_glyph","M 139 106 C 107 106 86 82 86 50 C 86 20 109 -5 139 -5 C 153 -5 165 -1 174 8 L 175 0 C 175 -63 154 -117 112 -160 C 105 -168 101 -174 101 -178 C 101 -188 105 -193 114 -193 C 123 -193 135 -181 152 -158 C 186 -110 203 -57 203 0 C 203 53 185 106 139 106 Z ",[75,82],{"id":83,"className":84,"d":85},"gl3Hi1a0C",[79],"M 751 53 C 751 83 725 106 695 106 C 664 106 638 83 638 53 C 638 23 664 0 695 0 C 725 0 751 23 751 53 Z M 475 53 C 475 83 449 106 419 106 C 388 106 363 83 363 53 C 363 23 388 0 419 0 C 449 0 475 23 475 53 Z M 200 53 C 200 83 174 106 143 106 C 113 106 87 83 87 53 C 87 23 113 0 143 0 C 173 0 200 23 200 53 Z ",[75,87],{"id":88,"className":89,"d":90},"gl3Hi1f0K",[79],"M 555 683 L 235 683 C 212 683 201 682 201 660 C 201 649 212 644 234 644 C 254 644 294 647 294 631 C 294 629 293 623 290 613 L 158 82 C 152 60 142 47 128 42 C 121 40 103 39 72 39 C 50 39 40 37 40 16 C 40 5 46 0 59 0 L 184 3 L 248 2 C 259 2 299 0 312 0 C 328 0 336 8 336 24 C 336 34 325 39 304 39 C 264 39 244 43 244 52 C 244 52 245 55 247 68 L 307 312 L 472 312 C 537 312 599 332 657 371 C 722 414 754 467 754 530 C 754 629 660 683 555 683 Z M 524 644 C 611 644 654 614 654 554 C 654 498 626 423 597 397 C 559 363 509 346 447 346 L 313 346 L 379 610 C 387 644 387 644 429 644 Z ",[75,92],{"id":93,"className":94,"d":95},"gl3Hi1R4L",[79],"M 527 373 C 527 419 482 442 432 442 C 389 442 355 419 329 373 C 308 419 273 442 222 442 C 173 442 133 419 101 374 C 74 335 60 306 60 287 C 60 278 65 273 75 273 C 84 273 90 278 92 287 C 111 345 153 413 220 413 C 253 413 269 392 269 351 C 269 330 251 252 216 118 C 199 51 169 18 126 18 C 112 18 99 21 88 26 C 114 36 127 54 127 80 C 127 106 114 119 87 119 C 54 119 29 91 29 58 C 29 12 76 -11 125 -11 C 167 -11 201 12 228 58 C 247 12 283 -11 335 -11 C 383 -11 423 12 455 57 C 482 96 496 125 496 144 C 496 153 491 158 481 158 C 472 158 467 153 464 144 C 447 87 402 18 337 18 C 304 18 287 38 287 79 C 287 92 292 120 303 165 L 337 300 C 356 375 387 413 431 413 C 445 413 458 410 469 405 C 442 396 429 378 429 351 C 429 325 443 312 470 312 C 502 312 527 341 527 373 Z ",[75,97],{"id":98,"className":99,"d":100},"gl3Hi1U0S",[79],"M 303 664 C 260 622 196 601 109 601 L 109 557 C 168 557 215 566 251 583 L 251 85 C 251 72 250 64 247 60 C 242 49 212 43 157 43 L 116 43 L 116 0 L 294 4 L 473 0 L 473 43 L 432 43 C 377 43 347 49 341 60 C 339 64 338 72 338 85 L 338 632 C 338 659 334 664 303 664 Z ",[75,102],{"id":103,"className":104,"d":105},"gl3Hi1cUT",[79],"M 393 627 L 259 87 C 255 72 251 63 248 59 C 239 48 205 43 147 43 C 114 43 98 45 98 18 C 98 6 105 0 119 0 C 173 0 233 5 289 4 L 378 3 C 404 3 443 2 462 0 C 479 0 488 9 488 26 C 488 35 483 41 472 43 L 444 43 C 395 43 366 45 359 50 C 356 52 355 55 355 60 C 355 65 356 72 358 82 L 489 601 C 496 624 494 630 516 632 C 519 633 538 633 575 633 C 632 633 669 630 685 625 C 717 614 729 599 729 550 C 729 523 726 493 720 460 C 720 447 727 441 741 441 C 746 441 751 442 754 445 C 759 455 762 463 763 469 L 790 658 C 790 670 779 676 756 676 L 148 676 C 120 676 118 674 111 653 L 48 478 C 47 473 43 462 43 459 C 43 447 50 441 64 441 C 74 441 82 450 88 467 C 122 560 144 619 247 631 C 258 632 284 633 325 633 L 367 633 C 385 633 393 635 393 627 Z ",[75,107],{"id":108,"className":109,"d":110},"gl3Hi1XEW",[79],"M 348 -297 C 357 -297 362 -292 362 -283 C 362 -278 360 -275 357 -272 C 301 -229 256 -157 222 -56 C 193 31 178 115 178 198 L 178 302 C 178 385 193 469 222 556 C 256 657 301 729 357 772 C 360 775 362 778 362 783 C 362 792 357 797 348 797 C 344 797 341 796 340 794 C 275 744 221 670 178 571 C 137 477 116 387 116 302 L 116 198 C 116 113 137 23 178 -71 C 221 -170 275 -244 340 -294 C 341 -296 344 -297 348 -297 Z ",[75,112],{"id":113,"className":114,"d":115},"gl3Hi1XIW",[79],"M 82 -294 C 147 -244 201 -170 244 -71 C 285 23 306 113 306 198 L 306 302 C 306 387 285 477 244 571 C 201 670 147 744 82 794 C 81 796 78 797 74 797 C 65 797 60 792 60 783 C 60 778 62 775 65 772 C 121 729 166 657 200 556 C 229 469 244 385 244 302 L 244 198 C 244 115 229 31 200 -56 C 166 -157 121 -229 65 -272 C 62 -275 60 -278 60 -283 C 60 -292 65 -297 74 -297 C 78 -297 81 -296 82 -294 Z ",[75,117],{"id":118,"className":119,"d":120},"gl3Hi1VgX",[79],"M 145 -314 C 145 -330 145 -351 166 -351 C 188 -351 189 -331 189 -314 L 189 814 C 189 830 188 851 167 851 C 145 851 145 831 145 814 Z ",[75,122],{"id":123,"className":124,"d":125},"gl3Hi1T8a",[79],"M 203 -10 C 252 -10 301 16 350 68 C 434 156 511 322 511 485 C 511 540 501 588 482 627 C 457 678 420 704 371 704 C 322 704 273 678 224 627 C 140 539 63 372 63 209 C 63 98 104 -10 203 -10 Z M 371 668 C 410 668 429 627 429 544 C 429 502 420 444 402 369 L 183 369 C 208 471 279 668 371 668 Z M 172 325 L 391 325 C 369 239 343 171 313 122 C 275 58 238 26 203 26 C 164 26 145 67 145 150 C 145 198 154 256 172 325 Z ",[75,127],{"id":128,"className":129,"d":130},"gl3Hi1Z4b",[79],"M 695 277 L 416 277 L 416 556 C 416 572 405 583 389 583 C 373 583 362 572 362 556 L 362 277 L 83 277 C 67 277 56 266 56 250 C 56 234 67 223 83 223 L 362 223 L 362 -56 C 362 -72 373 -83 389 -83 C 405 -83 416 -72 416 -56 L 416 223 L 695 223 C 711 223 722 234 722 250 C 722 263 708 277 695 277 Z ",[70,132],{"className":133},[134],"clip-path",[65,136],{"type":67},[138,139,146],"g",{"className":140,"transform":142,"dataTid":143,"dataPageWidth":144,"dataPageHeight":145},[141],"typst-page","translate(0, 0)","pd3ngfHc2OxEBN5Xvm\u002FBBqk","97","14",[138,147,151],{"dataTid":148,"className":149},"gd3ngfHc2OxEBN5Xvm\u002FBBqk",[150],"typst-group",[138,152,155],{"transform":153,"dataTid":154},"translate(0.000,0.000)","pJzA08\u002FujZwIgfbH9OBA7bM",[138,156,159,481],{"dataTid":157,"className":158},"gJzA08\u002FujZwIgfbH9OBA7bM",[150],[138,160,162],{"transform":153,"dataTid":161},"ptuuPFY2JPQsDCiSqLFo4oQ",[138,163,166,198,219,239,259,279,299,319,339,354,369,389,409,424,439,454,473],{"dataTid":164,"className":165},"gtuuPFY2JPQsDCiSqLFo4oQ",[150],[138,167,170],{"transform":168,"dataTid":169},"translate(0.000,9.361)","pE7Pe3cHqP3s99rC9ygpBBQ",[138,171,177,182],{"dataTid":172,"className":173,"fill":175,"transform":176},"gE7Pe3cHqP3s99rC9ygpBBQ",[174],"typst-text","#000","scale(0.011,-0.011)",[178,179],"use",{"x":180,"y":180,"href":181},"0","#gl3Hi1f0K",[138,183,185],{"transform":184},"scale(16,-16)",[186,187,191],"foreignObject",{"x":180,"y":188,"width":189,"height":190},"-50.38","48.88","62.50",[192,193,197],"h5:div",{"className":194,"style":196},[195],"tsel","font-size: 62px","𝑃",[138,199,202],{"transform":200,"dataTid":201},"translate(5.962,12.078)","peypDApLxG4E\u002F0mE7UpdL0U",[138,203,207,210],{"dataTid":204,"className":205,"fill":175,"transform":206},"geypDApLxG4E\u002F0mE7UpdL0U",[174],"scale(0.0077,-0.0077)",[178,208],{"x":180,"y":180,"href":209},"#gl3Hi1T8a",[138,211,212],{"transform":184},[186,213,215],{"x":180,"y":188,"width":214,"height":190},"34.00",[192,216,218],{"className":217,"style":196},[195],"𝜃",[138,220,223],{"transform":221,"dataTid":222},"translate(10.767,9.361)","p1Ae4He2LKUMdKNQbezvUlw",[138,224,227,230],{"dataTid":225,"className":226,"fill":175,"transform":176},"g1Ae4He2LKUMdKNQbezvUlw",[174],[178,228],{"x":180,"y":180,"href":229},"#gl3Hi1XEW",[138,231,232],{"transform":184},[186,233,235],{"x":180,"y":188,"width":234,"height":190},"26.38",[192,236,238],{"className":237,"style":196},[195],"(",[138,240,243],{"transform":241,"dataTid":242},"translate(15.409,9.361)","p9HA6gWESnEglcMwcua1hWM",[138,244,247,250],{"dataTid":245,"className":246,"fill":175,"transform":176},"g9HA6gWESnEglcMwcua1hWM",[174],[178,248],{"x":180,"y":180,"href":249},"#gl3Hi1R4L",[138,251,252],{"transform":184},[186,253,255],{"x":180,"y":188,"width":254,"height":190},"35.75",[192,256,258],{"className":257,"style":196},[195],"𝑥",[138,260,263],{"transform":261,"dataTid":262},"translate(21.701,12.078)","p45gwfYPR5FIiAOQYgbFw7o",[138,264,267,270],{"dataTid":265,"className":266,"fill":175,"transform":206},"g45gwfYPR5FIiAOQYgbFw7o",[174],[178,268],{"x":180,"y":180,"href":269},"#gl3Hi1cUT",[138,271,272],{"transform":184},[186,273,275],{"x":180,"y":188,"width":274,"height":190},"51.13",[192,276,278],{"className":277,"style":196},[195],"𝑇",[138,280,283],{"transform":281,"dataTid":282},"translate(27.999,12.078)","pdu\u002FuieOqPKkeq9\u002FR9IDqlY",[138,284,287,290],{"dataTid":285,"className":286,"fill":175,"transform":206},"gdu\u002FuieOqPKkeq9\u002FR9IDqlY",[174],[178,288],{"x":180,"y":180,"href":289},"#gl3Hi1Z4b",[138,291,292],{"transform":184},[186,293,295],{"x":180,"y":188,"width":294,"height":190},"48.63",[192,296,298],{"className":297,"style":196},[195],"+",[138,300,303],{"transform":301,"dataTid":302},"translate(33.990,12.078)","pxGw9J6YHZ5YbtDTZunemOc",[138,304,307,310],{"dataTid":305,"className":306,"fill":175,"transform":206},"gxGw9J6YHZ5YbtDTZunemOc",[174],[178,308],{"x":180,"y":180,"href":309},"#gl3Hi1U0S",[138,311,312],{"transform":184},[186,313,315],{"x":180,"y":188,"width":314,"height":190},"35.56",[192,316,318],{"className":317,"style":196},[195],"1",[138,320,323],{"transform":321,"dataTid":322},"translate(42.043,9.361)","pS6u9WMX6iYw\u002FuWSoWQHoOc",[138,324,327,330],{"dataTid":325,"className":326,"fill":175,"transform":176},"gS6u9WMX6iYw\u002FuWSoWQHoOc",[174],[178,328],{"x":180,"y":180,"href":329},"#gl3Hi1VgX",[138,331,332],{"transform":184},[186,333,335],{"x":180,"y":188,"width":334,"height":190},"20.81",[192,336,338],{"className":337,"style":196},[195],"|",[138,340,342],{"transform":341,"dataTid":242},"translate(48.761,9.361)",[138,343,345,347],{"dataTid":245,"className":344,"fill":175,"transform":176},[174],[178,346],{"x":180,"y":180,"href":249},[138,348,349],{"transform":184},[186,350,351],{"x":180,"y":188,"width":254,"height":190},[192,352,258],{"className":353,"style":196},[195],[138,355,357],{"transform":356,"dataTid":302},"translate(55.053,12.078)",[138,358,360,362],{"dataTid":305,"className":359,"fill":175,"transform":206},[174],[178,361],{"x":180,"y":180,"href":309},[138,363,364],{"transform":184},[186,365,366],{"x":180,"y":188,"width":314,"height":190},[192,367,318],{"className":368,"style":196},[195],[138,370,373],{"transform":371,"dataTid":372},"translate(60.051,9.361)","pUAb\u002F9lCbaBg2WDdv8AUOhk",[138,374,377,380],{"dataTid":375,"className":376,"fill":175,"transform":176},"gUAb\u002F9lCbaBg2WDdv8AUOhk",[174],[178,378],{"x":180,"y":180,"href":379},"#gl3Hi1Q0",[138,381,382],{"transform":184},[186,383,385],{"x":180,"y":188,"width":384,"height":190},"17.38",[192,386,388],{"className":387,"style":196},[195],",",[138,390,393],{"transform":391,"dataTid":392},"translate(64.942,9.361)","pgb3QkGi7QM03Kg+UM5VojU",[138,394,397,400],{"dataTid":395,"className":396,"fill":175,"transform":176},"ggb3QkGi7QM03Kg+UM5VojU",[174],[178,398],{"x":180,"y":180,"href":399},"#gl3Hi1a0C",[138,401,402],{"transform":184},[186,403,405],{"x":180,"y":188,"width":404,"height":190},"52.31",[192,406,408],{"className":407,"style":196},[195],"…",[138,410,412],{"transform":411,"dataTid":372},"translate(74.149,9.361)",[138,413,415,417],{"dataTid":375,"className":414,"fill":175,"transform":176},[174],[178,416],{"x":180,"y":180,"href":379},[138,418,419],{"transform":184},[186,420,421],{"x":180,"y":188,"width":384,"height":190},[192,422,388],{"className":423,"style":196},[195],[138,425,427],{"transform":426,"dataTid":242},"translate(79.040,9.361)",[138,428,430,432],{"dataTid":245,"className":429,"fill":175,"transform":176},[174],[178,431],{"x":180,"y":180,"href":249},[138,433,434],{"transform":184},[186,435,436],{"x":180,"y":188,"width":254,"height":190},[192,437,258],{"className":438,"style":196},[195],[138,440,442],{"transform":441,"dataTid":262},"translate(85.332,12.078)",[138,443,445,447],{"dataTid":265,"className":444,"fill":175,"transform":206},[174],[178,446],{"x":180,"y":180,"href":269},[138,448,449],{"transform":184},[186,450,451],{"x":180,"y":188,"width":274,"height":190},[192,452,278],{"className":453,"style":196},[195],[138,455,458],{"transform":456,"dataTid":457},"translate(92.247,9.361)","p2OdHd+uUQ\u002F4aDu3JIrLMZI",[138,459,462,465],{"dataTid":460,"className":461,"fill":175,"transform":176},"g2OdHd+uUQ\u002F4aDu3JIrLMZI",[174],[178,463],{"x":180,"y":180,"href":464},"#gl3Hi1XIW",[138,466,467],{"transform":184},[186,468,469],{"x":180,"y":188,"width":234,"height":190},[192,470,472],{"className":471,"style":196},[195],")",[138,474,476],{"transform":153,"dataTid":475},"p+nF9nphQ3U0gANvmKrhITw",[138,477],{"dataTid":478,"className":479,"dataHint":15},"g+nF9nphQ3U0gANvmKrhITw",[480],"typst-content-hint",[138,482,483],{"transform":153,"dataTid":475},[138,484],{"dataTid":478,"className":485,"dataHint":15},[480],[11,487,488,489,561,562,725,726,794,795,846],{},"Here ",[51,490,498,500,508,511,513],{"style":491,"className":492,"viewBox":493,"width":494,"height":495,"dataWidth":496,"dataHeight":497,"xmlns":61,"xmlnsXLink":62,"xmlns:h5":63},"vertical-align: -0.5445454545454546em;",[55],"0 0 6.000 19.000","0.5454545454545454em","1.7272727272727273em","6.000","19.000",[65,499,68],{"type":67},[70,501,503],{"className":502},[73],[75,504],{"id":505,"className":506,"d":507},"gl3Hi1aMN",[79],"M 164 -11 C 209 -11 256 16 302 70 C 382 163 455 335 455 498 C 455 539 448 582 434 625 C 416 678 382 705 333 705 C 288 705 243 678 196 624 C 116 532 42 360 42 196 C 42 94 75 -11 164 -11 Z M 332 675 C 365 675 382 637 382 560 C 382 517 372 452 352 367 L 155 367 C 180 460 204 529 229 574 C 266 641 301 675 332 675 Z M 145 327 L 341 327 C 322 246 299 181 274 132 C 235 57 199 19 164 19 C 131 19 115 58 115 135 C 115 185 125 249 145 327 Z ",[70,509],{"className":510},[134],[65,512],{"type":67},[138,514,519],{"className":515,"transform":142,"dataTid":516,"dataPageWidth":517,"dataPageHeight":518},[141],"plqRIdzaBgKAhDYAlkcekuE","6","19",[138,520,523],{"dataTid":521,"className":522},"glqRIdzaBgKAhDYAlkcekuE",[150],[138,524,526],{"transform":153,"dataTid":525},"pxY4rriuQcaESSNDbIL7nSA",[138,527,530,556],{"dataTid":528,"className":529},"gxY4rriuQcaESSNDbIL7nSA",[150],[138,531,533],{"transform":153,"dataTid":532},"pmZ6O0+j32fM61R2ahK1CYo",[138,534,537],{"dataTid":535,"className":536},"gmZ6O0+j32fM61R2ahK1CYo",[150],[138,538,541],{"transform":539,"dataTid":540},"translate(0.000,13.013)","pf8k1tDJ0hcA5Ck1Zkl5tL8",[138,542,545,548],{"dataTid":543,"className":544,"fill":175,"transform":176},"gf8k1tDJ0hcA5Ck1Zkl5tL8",[174],[178,546],{"x":180,"y":180,"href":547},"#gl3Hi1aMN",[138,549,550],{"transform":184},[186,551,553],{"x":180,"y":188,"width":552,"height":190},"30.19",[192,554,218],{"className":555,"style":196},[195],[138,557,558],{"transform":153,"dataTid":475},[138,559],{"dataTid":478,"className":560,"dataHint":15},[480]," means the model's weights, and ",[51,563,568,570,588,591,593],{"style":491,"className":564,"viewBox":565,"width":566,"height":495,"dataWidth":567,"dataHeight":497,"xmlns":61,"xmlnsXLink":62,"xmlns:h5":63},[55],"0 0 44.000 19.000","4em","44.000",[65,569,68],{"type":67},[70,571,573,576,579,582,585],{"className":572},[73],[75,574],{"id":77,"className":575,"d":80},[79],[75,577],{"id":83,"className":578,"d":85},[79],[75,580],{"id":93,"className":581,"d":95},[79],[75,583],{"id":98,"className":584,"d":100},[79],[75,586],{"id":103,"className":587,"d":105},[79],[70,589],{"className":590},[134],[65,592],{"type":67},[138,594,598],{"className":595,"transform":142,"dataTid":596,"dataPageWidth":597,"dataPageHeight":518},[141],"p2NLCIJs8FEETHILgE+Jh0M","44",[138,599,602],{"dataTid":600,"className":601},"g2NLCIJs8FEETHILgE+Jh0M",[150],[138,603,605],{"transform":153,"dataTid":604},"ptUnTFsLr1sIpfHO9sQocW4",[138,606,609,720],{"dataTid":607,"className":608},"gtUnTFsLr1sIpfHO9sQocW4",[150],[138,610,612],{"transform":153,"dataTid":611},"pD9eKovAfc7sGEHwUwxNnbo",[138,613,616,630,645,660,675,690,705],{"dataTid":614,"className":615},"gD9eKovAfc7sGEHwUwxNnbo",[150],[138,617,618],{"transform":539,"dataTid":242},[138,619,621,623],{"dataTid":245,"className":620,"fill":175,"transform":176},[174],[178,622],{"x":180,"y":180,"href":249},[138,624,625],{"transform":184},[186,626,627],{"x":180,"y":188,"width":254,"height":190},[192,628,258],{"className":629,"style":196},[195],[138,631,633],{"transform":632,"dataTid":302},"translate(6.292,15.730)",[138,634,636,638],{"dataTid":305,"className":635,"fill":175,"transform":206},[174],[178,637],{"x":180,"y":180,"href":309},[138,639,640],{"transform":184},[186,641,642],{"x":180,"y":188,"width":314,"height":190},[192,643,318],{"className":644,"style":196},[195],[138,646,648],{"transform":647,"dataTid":372},"translate(11.289,13.013)",[138,649,651,653],{"dataTid":375,"className":650,"fill":175,"transform":176},[174],[178,652],{"x":180,"y":180,"href":379},[138,654,655],{"transform":184},[186,656,657],{"x":180,"y":188,"width":384,"height":190},[192,658,388],{"className":659,"style":196},[195],[138,661,663],{"transform":662,"dataTid":392},"translate(16.181,13.013)",[138,664,666,668],{"dataTid":395,"className":665,"fill":175,"transform":176},[174],[178,667],{"x":180,"y":180,"href":399},[138,669,670],{"transform":184},[186,671,672],{"x":180,"y":188,"width":404,"height":190},[192,673,408],{"className":674,"style":196},[195],[138,676,678],{"transform":677,"dataTid":372},"translate(25.388,13.013)",[138,679,681,683],{"dataTid":375,"className":680,"fill":175,"transform":176},[174],[178,682],{"x":180,"y":180,"href":379},[138,684,685],{"transform":184},[186,686,687],{"x":180,"y":188,"width":384,"height":190},[192,688,388],{"className":689,"style":196},[195],[138,691,693],{"transform":692,"dataTid":242},"translate(30.279,13.013)",[138,694,696,698],{"dataTid":245,"className":695,"fill":175,"transform":176},[174],[178,697],{"x":180,"y":180,"href":249},[138,699,700],{"transform":184},[186,701,702],{"x":180,"y":188,"width":254,"height":190},[192,703,258],{"className":704,"style":196},[195],[138,706,708],{"transform":707,"dataTid":262},"translate(36.571,15.730)",[138,709,711,713],{"dataTid":265,"className":710,"fill":175,"transform":206},[174],[178,712],{"x":180,"y":180,"href":269},[138,714,715],{"transform":184},[186,716,717],{"x":180,"y":188,"width":274,"height":190},[192,718,278],{"className":719,"style":196},[195],[138,721,722],{"transform":153,"dataTid":475},[138,723],{"dataTid":478,"className":724,"dataHint":15},[480]," are the tokens already in the context. Generating ",[51,727,732,734,742,745,747],{"style":491,"className":728,"viewBox":729,"width":730,"height":495,"dataWidth":731,"dataHeight":497,"xmlns":61,"xmlnsXLink":62,"xmlns:h5":63},[55],"0 0 10.000 19.000","0.9090909090909091em","10.000",[65,733,68],{"type":67},[70,735,737],{"className":736},[73],[75,738],{"id":739,"className":740,"d":741},"gl3Hi1fsK",[79],"M 864 683 C 845 683 783 680 764 680 C 745 680 682 683 663 683 C 648 683 641 675 641 659 C 641 650 648 645 663 644 C 705 644 726 631 726 605 C 726 599 725 592 724 585 L 616 157 L 401 662 C 393 681 391 683 366 683 L 233 683 C 209 683 200 681 200 659 C 200 649 211 644 232 644 C 274 644 295 643 296 640 L 163 110 C 154 71 131 49 95 42 C 69 39 39 43 39 15 C 39 5 45 0 56 0 C 74 0 136 3 155 3 C 174 3 238 0 257 0 C 272 0 279 8 279 24 C 279 33 271 38 255 39 C 214 40 194 53 194 78 C 194 83 195 90 197 100 L 326 611 L 576 21 C 582 7 590 0 599 0 C 608 0 614 8 618 24 L 757 574 C 770 625 798 643 860 644 C 874 645 881 653 881 669 C 878 678 877 683 864 683 Z ",[70,743],{"className":744},[134],[65,746],{"type":67},[138,748,752],{"className":749,"transform":142,"dataTid":750,"dataPageWidth":751,"dataPageHeight":518},[141],"p4nRQulKUaKsIHoOarOlvdE","10",[138,753,756],{"dataTid":754,"className":755},"g4nRQulKUaKsIHoOarOlvdE",[150],[138,757,759],{"transform":153,"dataTid":758},"pXpsPljH6uIkv4w827107aM",[138,760,763,789],{"dataTid":761,"className":762},"gXpsPljH6uIkv4w827107aM",[150],[138,764,766],{"transform":153,"dataTid":765},"pnsHpVAjLR3IkeWOskK7elI",[138,767,770],{"dataTid":768,"className":769},"gnsHpVAjLR3IkeWOskK7elI",[150],[138,771,773],{"transform":539,"dataTid":772},"pBhaE0gjktdkunjkgG5XDx4",[138,774,777,780],{"dataTid":775,"className":776,"fill":175,"transform":176},"gBhaE0gjktdkunjkgG5XDx4",[174],[178,778],{"x":180,"y":180,"href":779},"#gl3Hi1fsK",[138,781,782],{"transform":184},[186,783,785],{"x":180,"y":188,"width":784,"height":190},"56.81",[192,786,788],{"className":787,"style":196},[195],"𝑁",[138,790,791],{"transform":153,"dataTid":475},[138,792],{"dataTid":478,"className":793,"dataHint":15},[480]," tokens takes ",[51,796,798,800,806,809,811],{"style":491,"className":797,"viewBox":729,"width":730,"height":495,"dataWidth":731,"dataHeight":497,"xmlns":61,"xmlnsXLink":62,"xmlns:h5":63},[55],[65,799,68],{"type":67},[70,801,803],{"className":802},[73],[75,804],{"id":739,"className":805,"d":741},[79],[70,807],{"className":808},[134],[65,810],{"type":67},[138,812,814],{"className":813,"transform":142,"dataTid":750,"dataPageWidth":751,"dataPageHeight":518},[141],[138,815,817],{"dataTid":754,"className":816},[150],[138,818,819],{"transform":153,"dataTid":758},[138,820,822,841],{"dataTid":761,"className":821},[150],[138,823,824],{"transform":153,"dataTid":765},[138,825,827],{"dataTid":768,"className":826},[150],[138,828,829],{"transform":539,"dataTid":772},[138,830,832,834],{"dataTid":775,"className":831,"fill":175,"transform":176},[174],[178,833],{"x":180,"y":180,"href":779},[138,835,836],{"transform":184},[186,837,838],{"x":180,"y":188,"width":784,"height":190},[192,839,788],{"className":840,"style":196},[195],[138,842,843],{"transform":153,"dataTid":475},[138,844],{"dataTid":478,"className":845,"dataHint":15},[480]," forward passes in the loop shown here. The runtime saves earlier work in a cache, so it does not need to process the whole prompt on every pass.",[11,848,849],{},[850,851],"img",{"alt":852,"src":853},"The chat template, 47 token IDs, the embedding, 28 blocks, the final norm, 128,256 logits and a greedy choice, with the new token fed back into the model","\u002Fassets\u002Fblog\u002Ffrom-safetensors-to-tokens-executing-a-transformer-step-by-step\u002Fpipeline.svg",[11,855,856],{},"Three kinds of data move through this loop. The parameters stay fixed for the whole run. The activations are the tensors each layer produces for this prompt. The KV cache keeps part of those activations, the keys and values of attention, so the next call doesn't have to compute the past again.",[11,858,859],{},"We use greedy decoding throughout this run. Each pass selects the token with the highest score.",[43,861,863],{"id":862},"turning-the-question-into-vectors","Turning the question into vectors",[11,865,866,867,871,872,875,876,879],{},"For this chat run, the question is wrapped in the markers Llama saw during training. A turn opens with ",[868,869,870],"code",{},"\u003C|start_header_id|>",", a role name and ",[868,873,874],{},"\u003C|end_header_id|>",". The user turn closes with ",[868,877,878],{},"\u003C|eot_id|>",", and the prompt ends by opening the assistant turn.",[11,881,882],{},"The model's chat template also adds a default system turn containing a knowledge cutoff and a date. We fixed the date to 27 Oct 2026 for this run. Keeping that date matters when reproducing the exact token IDs and probabilities.",[11,884,885,886,888,889,892,893,896],{},"The tokenizer then cuts this text into 47 pieces. Special tokens like ",[868,887,878],{}," have their own IDs above 128,000. Ordinary words are split into common pieces, most of them carrying their leading space. In Rust, the ",[868,890,891],{},"tokenizers"," crate reads the same ",[868,894,895],{},"tokenizer.json"," as every other runtime.",[898,899,904],"pre",{"className":900,"code":901,"language":902,"meta":903,"style":903},"language-rust shiki shiki-themes github-dark","use tokenizers::Tokenizer;\n\nlet tokenizer = Tokenizer::from_file(\"tokenizer.json\")?;\nlet question = \"What is the capital of France? Answer in one word.\";\nlet prompt = format!(\n    \"\u003C|begin_of_text|>\u003C|start_header_id|>system\u003C|end_header_id|>\\n\\n\\\n     Cutting Knowledge Date: December 2023\\nToday Date: 27 Oct 2026\\n\\n\u003C|eot_id|>\\\n     \u003C|start_header_id|>user\u003C|end_header_id|>\\n\\n{question}\u003C|eot_id|>\\\n     \u003C|start_header_id|>assistant\u003C|end_header_id|>\\n\\n\"\n);\nlet encoding = tokenizer.encode(prompt, false)?;\n\u002F\u002F 47 IDs, starting with [128000, 128006, 9125, 128007, ...]\n","rust","",[868,905,906,928,935,968,983,999,1012,1029,1040,1051,1057,1088],{"__ignoreMap":903},[907,908,911,914,918,921,924],"span",{"class":909,"line":910},"line",1,[907,912,178],{"class":913},"snl16",[907,915,917],{"class":916},"svObZ"," tokenizers",[907,919,920],{"class":913},"::",[907,922,923],{"class":916},"Tokenizer",[907,925,927],{"class":926},"s95oV",";\n",[907,929,931],{"class":909,"line":930},2,[907,932,934],{"emptyLinePlaceholder":933},true,"\n",[907,936,938,941,944,947,950,952,955,957,961,963,966],{"class":909,"line":937},3,[907,939,940],{"class":913},"let",[907,942,943],{"class":926}," tokenizer ",[907,945,946],{"class":913},"=",[907,948,949],{"class":916}," Tokenizer",[907,951,920],{"class":913},[907,953,954],{"class":916},"from_file",[907,956,238],{"class":926},[907,958,960],{"class":959},"sU2Wk","\"tokenizer.json\"",[907,962,472],{"class":926},[907,964,965],{"class":913},"?",[907,967,927],{"class":926},[907,969,971,973,976,978,981],{"class":909,"line":970},4,[907,972,940],{"class":913},[907,974,975],{"class":926}," question ",[907,977,946],{"class":913},[907,979,980],{"class":959}," \"What is the capital of France? Answer in one word.\"",[907,982,927],{"class":926},[907,984,986,988,991,993,996],{"class":909,"line":985},5,[907,987,940],{"class":913},[907,989,990],{"class":926}," prompt ",[907,992,946],{"class":913},[907,994,995],{"class":916}," format!",[907,997,998],{"class":926},"(\n",[907,1000,1002,1005,1009],{"class":909,"line":1001},6,[907,1003,1004],{"class":959},"    \"\u003C|begin_of_text|>\u003C|start_header_id|>system\u003C|end_header_id|>",[907,1006,1008],{"class":1007},"sDLfK","\\n\\n",[907,1010,1011],{"class":959},"\\\n",[907,1013,1015,1018,1021,1024,1026],{"class":909,"line":1014},7,[907,1016,1017],{"class":959},"     Cutting Knowledge Date: December 2023",[907,1019,1020],{"class":1007},"\\n",[907,1022,1023],{"class":959},"Today Date: 27 Oct 2026",[907,1025,1008],{"class":1007},[907,1027,1028],{"class":959},"\u003C|eot_id|>\\\n",[907,1030,1032,1035,1037],{"class":909,"line":1031},8,[907,1033,1034],{"class":959},"     \u003C|start_header_id|>user\u003C|end_header_id|>",[907,1036,1008],{"class":1007},[907,1038,1039],{"class":959},"{question}\u003C|eot_id|>\\\n",[907,1041,1043,1046,1048],{"class":909,"line":1042},9,[907,1044,1045],{"class":959},"     \u003C|start_header_id|>assistant\u003C|end_header_id|>",[907,1047,1008],{"class":1007},[907,1049,1050],{"class":959},"\"\n",[907,1052,1054],{"class":909,"line":1053},10,[907,1055,1056],{"class":926},");\n",[907,1058,1060,1062,1065,1067,1070,1073,1076,1079,1082,1084,1086],{"class":909,"line":1059},11,[907,1061,940],{"class":913},[907,1063,1064],{"class":926}," encoding ",[907,1066,946],{"class":913},[907,1068,1069],{"class":926}," tokenizer",[907,1071,1072],{"class":913},".",[907,1074,1075],{"class":916},"encode",[907,1077,1078],{"class":926},"(prompt, ",[907,1080,1081],{"class":1007},"false",[907,1083,472],{"class":926},[907,1085,965],{"class":913},[907,1087,927],{"class":926},[907,1089,1091],{"class":909,"line":1090},12,[907,1092,1094],{"class":1093},"sAwPA","\u002F\u002F 47 IDs, starting with [128000, 128006, 9125, 128007, ...]\n",[11,1096,1097],{},"Only 12 of the 47 tokens come from the question. The other 35 come from the template. All 47 pass through every layer, including the role markers and dates.",[11,1099,1100],{},[850,1101],{"alt":1102,"src":1103},"The 47 tokens of the formatted prompt, split into special tokens, template text and the user's question, then the lookup in the embedding table","\u002Fassets\u002Fblog\u002Ffrom-safetensors-to-tokens-executing-a-transformer-step-by-step\u002Fembedding.svg",[11,1105,1106],{},"Each ID now picks one row of the embedding table. The table has 128,256 rows, one per vocabulary entry, and each row holds 3072 numbers. Foundry uses a gather operation to select these rows.",[898,1108,1110],{"className":900,"code":1109,"language":902,"meta":903,"style":903},"let flat_ids = input_ids.reshape(Shape::try_from([input_ids.shape().numel()])?)?;\nlet embeds = self.weight.index_select(device, 0, &flat_ids)?;\n",[868,1111,1112,1166],{"__ignoreMap":903},[907,1113,1114,1116,1119,1121,1124,1126,1129,1131,1134,1136,1139,1142,1144,1147,1150,1152,1155,1158,1160,1162,1164],{"class":909,"line":910},[907,1115,940],{"class":913},[907,1117,1118],{"class":926}," flat_ids ",[907,1120,946],{"class":913},[907,1122,1123],{"class":926}," input_ids",[907,1125,1072],{"class":913},[907,1127,1128],{"class":916},"reshape",[907,1130,238],{"class":926},[907,1132,1133],{"class":916},"Shape",[907,1135,920],{"class":913},[907,1137,1138],{"class":916},"try_from",[907,1140,1141],{"class":926},"([input_ids",[907,1143,1072],{"class":913},[907,1145,1146],{"class":916},"shape",[907,1148,1149],{"class":926},"()",[907,1151,1072],{"class":913},[907,1153,1154],{"class":916},"numel",[907,1156,1157],{"class":926},"()])",[907,1159,965],{"class":913},[907,1161,472],{"class":926},[907,1163,965],{"class":913},[907,1165,927],{"class":926},[907,1167,1168,1170,1173,1175,1178,1180,1183,1185,1188,1191,1193,1196,1199,1202,1204],{"class":909,"line":930},[907,1169,940],{"class":913},[907,1171,1172],{"class":926}," embeds ",[907,1174,946],{"class":913},[907,1176,1177],{"class":1007}," self",[907,1179,1072],{"class":913},[907,1181,1182],{"class":926},"weight",[907,1184,1072],{"class":913},[907,1186,1187],{"class":916},"index_select",[907,1189,1190],{"class":926},"(device, ",[907,1192,180],{"class":1007},[907,1194,1195],{"class":926},", ",[907,1197,1198],{"class":913},"&",[907,1200,1201],{"class":926},"flat_ids)",[907,1203,965],{"class":913},[907,1205,927],{"class":926},[11,1207,1208,1209,1273,1274,1277],{},"The result is a tensor ",[51,1210,1212,1214,1222,1225,1227],{"style":491,"className":1211,"viewBox":729,"width":730,"height":495,"dataWidth":731,"dataHeight":497,"xmlns":61,"xmlnsXLink":62,"xmlns:h5":63},[55],[65,1213,68],{"type":67},[70,1215,1217],{"className":1216},[73],[75,1218],{"id":1219,"className":1220,"d":1221},"gl3Hi1QUL",[79],"M 834 683 C 817 683 758 680 741 680 C 721 680 651 683 631 683 C 616 683 608 675 608 659 C 608 650 613 645 624 644 C 648 641 660 633 660 618 C 660 609 654 598 642 585 L 486 417 C 457 485 429 552 400 620 C 401 622 402 624 405 626 C 414 636 431 642 455 644 C 471 646 479 654 479 667 C 479 678 473 683 460 683 C 436 683 359 680 335 680 C 314 680 245 683 224 683 C 209 683 202 675 202 659 C 202 649 211 644 229 644 C 274 644 288 644 300 615 L 416 342 L 208 119 C 205 116 201 112 196 109 C 155 66 108 42 53 39 C 36 38 27 29 27 14 C 30 5 31 0 44 0 C 61 0 120 3 137 3 C 158 3 227 0 248 0 C 263 0 270 8 270 24 C 270 33 265 38 254 39 C 230 41 218 50 218 65 C 218 75 227 89 244 107 C 306 173 369 239 430 306 C 464 225 499 145 532 63 C 531 61 530 59 527 56 C 518 47 502 42 478 39 C 463 37 455 29 455 16 C 455 5 461 0 474 0 L 599 3 C 620 3 690 0 708 0 C 723 0 731 8 731 23 C 731 33 724 38 709 39 C 678 39 658 40 651 44 C 644 48 636 60 628 80 L 501 383 C 569 455 620 509 653 546 C 677 571 693 588 702 595 C 739 626 780 642 825 644 C 846 645 851 650 851 669 C 848 678 847 683 834 683 Z ",[70,1223],{"className":1224},[134],[65,1226],{"type":67},[138,1228,1231],{"className":1229,"transform":142,"dataTid":1230,"dataPageWidth":751,"dataPageHeight":518},[141],"pUirdjqd5lOcqYYCohGo69w",[138,1232,1235],{"dataTid":1233,"className":1234},"gUirdjqd5lOcqYYCohGo69w",[150],[138,1236,1238],{"transform":153,"dataTid":1237},"pQh\u002FzPXQ8oRY5nZf5tKBD5M",[138,1239,1242,1268],{"dataTid":1240,"className":1241},"gQh\u002FzPXQ8oRY5nZf5tKBD5M",[150],[138,1243,1245],{"transform":153,"dataTid":1244},"pZpZaRnOPtOQFf6WL3elTk8",[138,1246,1249],{"dataTid":1247,"className":1248},"gZpZaRnOPtOQFf6WL3elTk8",[150],[138,1250,1252],{"transform":539,"dataTid":1251},"perUXa3sJ\u002Fl4mdR6DRzbJZU",[138,1253,1256,1259],{"dataTid":1254,"className":1255,"fill":175,"transform":176},"gerUXa3sJ\u002Fl4mdR6DRzbJZU",[174],[178,1257],{"x":180,"y":180,"href":1258},"#gl3Hi1QUL",[138,1260,1261],{"transform":184},[186,1262,1264],{"x":180,"y":188,"width":1263,"height":190},"54.94",[192,1265,1267],{"className":1266,"style":196},[195],"𝑋",[138,1269,1270],{"transform":153,"dataTid":475},[138,1271],{"dataTid":478,"className":1272,"dataHint":15},[480]," of shape ",[868,1275,1276],{},"[1, 47, 3072]",". That means one prompt, 47 positions and 3072 numbers per position. This width is called the hidden size. It stays at 3072 between blocks, though some operations inside a block use wider or narrower vectors.",[43,1279,1281],{"id":1280},"inside-a-transformer-block","Inside a Transformer block",[11,1283,1284],{},"A block changes the vectors in two steps. First, attention lets each position read from itself and earlier positions. Then a feed-forward network, the MLP, works on each position alone. Both steps start with a normalization and end by adding their result back to their input. This running sum is called the residual stream.",[51,1286,1293,1295,1391,1394,1396],{"style":53,"className":1287,"viewBox":1288,"width":1289,"height":1290,"dataWidth":1291,"dataHeight":1292,"xmlns":61,"xmlnsXLink":62,"xmlns:h5":63},[55],"0 0 218.000 11.000","19.818181818181817em","1em","218.000","11.000",[65,1294,68],{"type":67},[70,1296,1298,1303,1308,1313,1318,1323,1328,1333,1338,1343,1348,1353,1358,1363,1368,1373,1378,1383,1388],{"className":1297},[73],[75,1299],{"id":1300,"className":1301,"d":1302},"gl3Hi1Qk",[79],"M 318 -248 C 327 -248 332 -243 332 -234 C 332 -231 330 -227 327 -223 C 275 -183 233 -117 202 -26 C 175 53 161 131 161 208 L 161 292 C 161 369 175 447 202 526 C 233 617 275 683 327 723 C 330 726 332 730 332 734 C 332 743 327 748 318 748 C 317 748 314 747 311 745 C 251 699 201 631 160 540 C 121 453 101 371 101 292 L 101 208 C 101 129 121 47 160 -40 C 201 -131 251 -199 311 -245 C 314 -247 317 -248 318 -248 Z ",[75,1304],{"id":1305,"className":1306,"d":1307},"gl3Hi1Qo",[79],"M 78 -245 C 138 -199 188 -131 229 -40 C 268 47 288 129 288 208 L 288 292 C 288 371 268 453 229 540 C 188 631 138 699 78 745 C 75 747 72 748 71 748 C 62 748 57 743 57 734 C 57 730 59 726 62 723 C 114 683 156 617 187 526 C 214 447 228 369 228 292 L 228 208 C 228 131 214 53 187 -26 C 156 -117 114 -183 62 -223 C 59 -227 57 -231 57 -234 C 57 -243 62 -248 71 -248 C 72 -248 75 -247 78 -245 Z ",[75,1309],{"id":1310,"className":1311,"d":1312},"gl3Hi1Qw",[79],"M 698 274 L 413 274 L 413 559 C 413 575 405 583 389 583 C 373 583 365 575 365 559 L 365 274 L 80 274 C 64 274 56 266 56 250 C 56 234 64 226 80 226 L 365 226 L 365 -59 C 365 -75 373 -83 389 -83 C 405 -83 413 -75 413 -59 L 413 226 L 698 226 C 714 226 722 234 722 250 C 722 263 711 274 698 274 Z ",[75,1314],{"id":1315,"className":1316,"d":1317},"gl3Hi1R4",[79],"M 698 367 L 80 367 C 64 367 56 359 56 344 C 56 329 64 321 80 321 L 698 321 C 714 321 722 329 722 344 C 722 356 711 367 698 367 Z M 698 179 L 80 179 C 64 179 56 171 56 156 C 56 141 64 133 80 133 L 698 133 C 714 133 722 141 722 156 C 722 169 711 179 698 179 Z ",[75,1319],{"id":1320,"className":1321,"d":1322},"gl3Hi1SI",[79],"M 583 3 L 717 0 L 717 39 L 695 39 C 670 39 653 40 646 42 C 631 47 621 58 615 75 L 401 694 C 397 707 390 716 375 716 C 362 716 353 709 348 694 L 143 102 C 129 61 92 39 32 38 L 32 0 L 137 3 L 254 0 L 254 39 C 205 39 180 54 180 85 C 180 92 181 96 182 98 L 227 225 L 465 225 L 516 77 C 518 71 519 67 519 64 C 519 47 492 39 437 39 L 437 0 C 479 3 528 4 583 3 Z M 345 569 L 451 264 L 240 264 Z ",[75,1324],{"id":1325,"className":1326,"d":1327},"gl3Hi1S4",[79],"M 175 108 L 175 622 L 407 24 C 413 8 421 0 430 0 C 439 0 446 7 452 21 L 689 630 L 689 81 C 689 63 685 51 678 46 C 671 41 650 39 615 39 L 587 39 L 587 0 C 614 2 663 3 733 3 C 803 3 852 2 879 0 L 879 39 L 851 39 C 816 39 795 41 788 46 C 781 51 777 63 777 81 L 777 602 C 777 620 781 631 788 636 C 795 641 816 644 851 644 L 879 644 L 879 683 L 708 683 C 681 683 679 682 671 662 L 458 115 L 247 659 C 238 679 235 683 208 683 L 37 683 L 37 644 L 65 644 C 100 644 121 641 128 636 C 135 631 139 620 139 602 L 139 108 C 139 91 135 77 127 64 C 116 47 86 39 37 39 L 37 0 L 157 3 L 277 0 L 277 39 C 204 39 175 51 175 108 Z ",[75,1329],{"id":1330,"className":1331,"d":1332},"gl3Hi1S8",[79],"M 576 15 C 585.56964 0.9519348 589 0 596 0 C 613 0 614 7 614 30 L 614 575 C 614 592 618 606 626 619 C 637 636 667 644 716 644 L 716 683 L 596 680 L 477 683 L 477 644 C 526 644 556 636 567 619 C 575 606 579 592 579 575 L 579 166 L 237 668 C 230 678 220 683 205 683 L 33 683 L 33 644 L 66 644 C 106 644 128 642 133 638 C 134 637 135 632 135 624 L 135 108 C 135 91 131 77 123 64 C 112 47 82 39 33 39 L 33 0 L 152 3 L 272 0 L 272 39 C 223 39 193 47 182 64 C 174 77 170 91 170 108 L 170 611 Z ",[75,1334],{"id":1335,"className":1336,"d":1337},"gl3Hi1TM",[79],"M 611 501 C 611 558 580 604 519 639 C 468 668 411 683 349 683 L 35 683 L 35 644 L 63 644 C 98 644 119 641 126 636 C 133 631 137 620 137 602 L 137 81 C 137 63 133 51 126 46 C 119 41 98 38 63 38 L 35 38 L 35 -1 C 62 2 111 3 183 3 C 256 3 305 2 332 -1 L 332 38 L 304 38 C 269 38 248 41 241 46 C 234 51 230 63 230 81 L 230 327 L 340 327 C 404 327 451 289 462 235 C 464 224 465 200 465 165 C 465 138 466 118 467 106 C 473 22 546 -22 634 -22 C 672 -22 699 -6 714 25 C 726 48 732 70 732 91 C 732 104 726 111 715 111 C 706 111 700 105 699 92 C 694 36 673 8 638 8 C 603 8 591 41 585 70 C 579 104 575 130 572 147 L 559 226 C 544 278 507 316 448 339 C 524 361 611 415 611 501 Z M 470 600 C 491 581 502 548 502 501 C 502 463 492 431 473 404 C 450 373 405 357 336 357 L 230 357 L 230 609 C 230 630 236 641 248 642 C 255 643 274 644 306 644 C 385 644 426 641 470 600 Z ",[75,1339],{"id":1340,"className":1341,"d":1342},"gl3Hi1TQ",[79],"M 252 669 C 359 669 422 601 440 466 C 441 456 447 451 456 451 C 467 451 472 460 472 478 L 472 677 C 472 696 467 705 458 705 C 452 705 446 700 440 690 L 408 638 C 364 680 324 705 251 705 C 142 705 56 617 56 509 C 56 417 116 349 187 324 L 317 292 C 384 275 428 229 428 154 C 428 79 376 17 301 17 C 170 17 93 91 88 218 C 88 227 83 232 73 232 C 62 232 56 223 56 204 L 56 6 C 56 -13 61 -22 71 -22 C 77 -22 83 -17 89 -7 L 121 45 C 164 0 224 -22 302 -22 C 359 -22 407 -1 444 40 C 481 81 499 130 499 188 C 499 235 485 278 458 317 C 431 356 394 380 349 391 L 223 422 C 170 434 127 481 127 544 C 127 614 182 669 252 669 Z ",[75,1344],{"id":1345,"className":1346,"d":1347},"gl3Hi1UY",[79],"M 415 251 C 415 366 348 448 236 448 C 176 448 126 425 85 378 C 47 334 28 281 28 220 C 28 157 49 104 91 58 C 133 12 185 -11 247 -11 C 304 -11 348 9 378 48 C 403 81 415 105 415 121 C 415 131 409 136 398 136 C 390 136 385 131 382 120 C 360 54 317 21 253 21 C 208 21 173 42 146 85 C 127 116 118 163 117 227 L 387 227 C 409 227 415 229 415 251 Z M 326 349 C 337 316 342 284 342 255 L 118 255 C 121 321 138 366 168 389 C 193 409 215 419 236 419 C 281 419 311 396 326 349 Z ",[75,1349],{"id":1350,"className":1351,"d":1352},"gl3Hi1Uo",[79],"M 194 601 C 194 631 169 657 139 657 C 109 657 83 631 83 601 C 83 571 108 544 138 544 C 169 544 194 570 194 601 Z M 143 3 L 247 0 L 247 39 C 214 39 194 41 188 45 C 182 49 180 60 180 78 L 180 445 L 37 433 L 37 395 C 70 395 91 392 98 387 C 105 382 108 368 108 345 L 108 79 C 108 60 105 48 98 44 C 91 40 69 39 33 39 L 33 0 Z ",[75,1354],{"id":1355,"className":1356,"d":1357},"gl3Hi1U4",[79],"M 315 413 C 361 413 384 378 384 307 L 384 79 C 384 60 381 48 374 44 C 367 40 344 38 307 38 L 307 0 L 423 3 L 538 0 L 538 38 C 501 38 479 40 472 44 C 465 48 461 60 461 79 L 461 259 C 461 339 513 413 590 413 C 637 413 660 378 660 307 L 660 79 C 660 60 656 48 649 44 C 642 40 619 38 582 38 L 582 0 L 698 3 L 813 0 L 813 38 C 781 38 760 39 750 42 C 740 45 736 52 736 64 L 736 251 C 736 298 734 331 731 350 C 721 411 676 442 597 442 C 534 442 487 413 455 354 C 441 413 397 442 322 442 C 259 442 211 412 179 352 L 179 442 L 32 431 L 32 393 C 68 393 90 390 98 384 C 106 378 109 365 109 342 L 109 79 C 109 60 105 48 98 44 C 91 40 69 38 32 38 L 32 0 L 148 3 L 263 0 L 263 38 C 226 38 203 40 196 44 C 189 48 185 60 185 79 L 185 259 C 185 340 238 413 315 413 Z ",[75,1359],{"id":1360,"className":1361,"d":1362},"gl3Hi1U8",[79],"M 314 413 C 359 413 382 378 382 307 L 382 79 C 382 60 379 48 372 44 C 365 40 343 38 306 38 L 306 0 L 421 3 L 535 0 L 535 38 C 503 38 482 39 472 42 C 462 45 458 52 458 64 L 458 251 C 458 298 457 331 454 350 C 444 411 399 442 320 442 C 257 442 210 412 179 352 L 179 442 L 32 431 L 32 393 C 68 393 90 390 98 384 C 106 378 109 365 109 342 L 109 79 C 109 60 105 48 98 44 C 91 40 69 38 32 38 L 32 0 L 147 3 L 261 0 L 261 38 C 224 38 203 40 196 44 C 189 48 185 60 185 79 L 185 259 C 185 340 236 413 314 413 Z ",[75,1364],{"id":1365,"className":1366,"d":1367},"gl3Hi1VA",[79],"M 249 -11 C 311 -11 363 11 406 55 C 449 99 471 152 471 214 C 471 277 450 332 408 378 C 366 424 313 448 250 448 C 187 448 135 424 92 378 C 49 332 28 277 28 214 C 28 152 49 99 92 55 C 135 11 188 -11 249 -11 Z M 250 21 C 202 21 166 42 141 85 C 125 113 117 159 117 222 C 117 283 125 327 140 355 C 164 398 200 419 249 419 C 296 419 332 398 357 357 C 374 329 382 284 382 222 C 382 106 348 21 250 21 Z ",[75,1369],{"id":1370,"className":1371,"d":1372},"gl3Hi1VM",[79],"M 364 378 C 364 417 327 442 288 442 C 237 442 198 412 172 351 L 172 442 L 28 431 L 28 393 C 63 393 84 390 92 384 C 100 378 104 365 104 342 L 104 79 C 104 60 101 48 94 44 C 87 40 65 38 28 38 L 28 0 L 143 3 C 185 4 228 3 271 0 L 271 38 L 247 38 C 214 38 193 41 186 46 C 179 51 176 63 176 81 L 176 232 C 176 275 184 314 199 347 C 219 390 248 412 287 413 C 277 403 272 391 272 377 C 272 346 287 331 318 331 C 345 331 364 352 364 378 Z ",[75,1374],{"id":1375,"className":1376,"d":1377},"gl3Hi1VU",[79],"M 332 126 L 332 186 L 300 186 L 300 128 C 300 78 282 22 238 22 C 197 22 177 56 177 124 L 177 395 L 316 395 L 316 433 L 177 433 L 177 615 L 145 615 C 144 582 141 553 134 527 C 117 461 78 427 19 424 L 19 395 L 102 395 L 102 126 C 102 67 120 29 155 10 C 182 -4 207 -11 232 -11 C 298 -11 332 55 332 126 Z ",[75,1379],{"id":1380,"className":1381,"d":1382},"gl3Hi1fUK",[79],"M 881 668 C 881 678 875 683 863 683 L 736 680 L 609 683 C 593 683 586 674 586 659 C 586 652 589 647 594 646 C 604 645 612 644 617 644 C 648 643 665 641 670 640 C 675 639 678 636 678 631 C 677 628 676 622 674 613 L 615 375 L 321 375 L 379 602 C 384 623 393 636 406 641 C 413 643 430 644 458 644 C 485 644 496 644 496 668 C 496 678 490 683 478 683 L 351 680 L 223 683 C 207 683 200 674 200 659 C 200 652 203 647 209 646 C 219 645 227 644 232 644 C 263 643 281 641 286 640 C 291 639 293 636 293 631 C 293 629 292 623 289 613 L 156 82 C 151 60 141 47 126 42 C 119 40 101 39 70 39 C 48 39 39 37 39 15 C 39 5 45 0 57 0 L 183 3 L 246 2 C 257 2 299 0 310 0 C 326 0 334 8 334 24 C 334 34 323 39 302 39 C 262 39 242 43 242 52 C 242 52 243 55 245 68 L 312 336 L 605 336 L 538 68 C 535 53 523 43 503 40 C 498 39 481 39 452 39 C 433 39 424 31 424 15 C 424 5 430 0 442 0 L 568 3 L 631 2 C 642 2 683 0 696 0 C 712 0 720 8 720 24 C 720 34 709 39 688 39 C 647 39 627 43 627 52 C 627 52 628 55 630 68 L 764 602 C 769 623 778 636 791 641 C 797 643 814 644 843 644 C 869 644 881 644 881 668 Z ",[75,1384],{"id":1385,"className":1386,"d":1387},"gl3Hi1QIL",[79],"M 642 680 C 623 680 559 683 540 683 C 525 683 518 675 518 659 C 518 650 525 645 539 644 C 582 644 603 631 603 605 C 603 599 602 592 600 585 L 511 232 C 496 175 467 126 424 84 C 377 39 325 17 269 17 C 195 17 152 67 152 141 L 153 150 C 153 169 156 191 163 217 L 259 602 C 264 623 273 636 286 641 C 292 643 309 644 338 644 C 364 644 375 644 375 668 C 375 678 369 683 358 683 L 231 680 L 104 683 C 88 683 81 674 81 659 C 81 652 84 647 89 646 C 99 645 107 644 112 644 C 143 643 161 641 166 640 C 171 639 173 636 173 631 C 173 629 172 622 169 611 L 74 230 C 69 211 67 192 67 172 C 67 57 150 -22 265 -22 C 330 -22 390 3 445 54 C 498 102 532 158 548 223 L 636 574 C 649 625 677 643 739 644 C 753 645 760 653 760 668 L 760 672 C 757 679 752 683 743 683 C 724 683 661 680 642 680 Z ",[75,1389],{"id":1219,"className":1390,"d":1221},[79],[70,1392],{"className":1393},[134],[65,1395],{"type":67},[138,1397,1402],{"className":1398,"transform":142,"dataTid":1399,"dataPageWidth":1400,"dataPageHeight":1401},[141],"p\u002F85x18KJOOwmB+\u002FBZb7gec","218","11",[138,1403,1406],{"dataTid":1404,"className":1405},"g\u002F85x18KJOOwmB+\u002FBZb7gec",[150],[138,1407,1409],{"transform":153,"dataTid":1408},"pt15obKlFVbY5jCIBhLLe04",[138,1410,1413,1720],{"dataTid":1411,"className":1412},"gt15obKlFVbY5jCIBhLLe04",[150],[138,1414,1416],{"transform":153,"dataTid":1415},"pyWaBVTEplOQOV0vYRF8wJ0",[138,1417,1420,1439,1458,1502,1521,1536,1554,1574,1589,1604,1622,1670,1685,1700,1715],{"dataTid":1418,"className":1419},"gyWaBVTEplOQOV0vYRF8wJ0",[150],[138,1421,1424],{"transform":1422,"dataTid":1423},"translate(0.000,8.228)","pzflOl9Rc+LkJgayB5QZbX8",[138,1425,1428,1431],{"dataTid":1426,"className":1427,"fill":175,"transform":176},"gzflOl9Rc+LkJgayB5QZbX8",[174],[178,1429],{"x":180,"y":180,"href":1430},"#gl3Hi1fUK",[138,1432,1433],{"transform":184},[186,1434,1435],{"x":180,"y":188,"width":784,"height":190},[192,1436,1438],{"className":1437,"style":196},[195],"𝐻",[138,1440,1443],{"transform":1441,"dataTid":1442},"translate(13.055,8.228)","pTVgzH5qlGVoRBbtwYAF8r4",[138,1444,1447,1450],{"dataTid":1445,"className":1446,"fill":175,"transform":176},"gTVgzH5qlGVoRBbtwYAF8r4",[174],[178,1448],{"x":180,"y":180,"href":1449},"#gl3Hi1R4",[138,1451,1452],{"transform":184},[186,1453,1455],{"x":180,"y":188,"width":1454,"height":190},"48.62",[192,1456,946],{"className":1457,"style":196},[195],[138,1459,1462],{"transform":1460,"dataTid":1461},"translate(24.668,8.228)","peA4ouKc3tuIpH9RV8V+UG4",[138,1463,1466,1469,1473,1477,1481,1485,1489,1493],{"dataTid":1464,"className":1465,"fill":175,"transform":176},"geA4ouKc3tuIpH9RV8V+UG4",[174],[178,1467],{"x":180,"y":180,"href":1468},"#gl3Hi1TM",[178,1470],{"x":1471,"y":180,"href":1472},"736","#gl3Hi1S4",[178,1474],{"x":1475,"y":180,"href":1476},"1653","#gl3Hi1TQ",[178,1478],{"x":1479,"y":180,"href":1480},"2209","#gl3Hi1S8",[178,1482],{"x":1483,"y":180,"href":1484},"2959","#gl3Hi1VA",[178,1486],{"x":1487,"y":180,"href":1488},"3459","#gl3Hi1VM",[178,1490],{"x":1491,"y":180,"href":1492},"3851","#gl3Hi1U4",[138,1494,1495],{"transform":184},[186,1496,1498],{"x":180,"y":188,"width":1497,"height":190},"292.75",[192,1499,1501],{"className":1500,"style":196},[195],"RMSNorm",[138,1503,1506],{"transform":1504,"dataTid":1505},"translate(76.192,8.228)","psVZyKwl7TJ0WP15eV58nhs",[138,1507,1510,1513],{"dataTid":1508,"className":1509,"fill":175,"transform":176},"gsVZyKwl7TJ0WP15eV58nhs",[174],[178,1511],{"x":180,"y":180,"href":1512},"#gl3Hi1Qk",[138,1514,1515],{"transform":184},[186,1516,1518],{"x":180,"y":188,"width":1517,"height":190},"24.31",[192,1519,238],{"className":1520,"style":196},[195],[138,1522,1524],{"transform":1523,"dataTid":1251},"translate(80.471,8.228)",[138,1525,1527,1529],{"dataTid":1254,"className":1526,"fill":175,"transform":176},[174],[178,1528],{"x":180,"y":180,"href":1258},[138,1530,1531],{"transform":184},[186,1532,1533],{"x":180,"y":188,"width":1263,"height":190},[192,1534,1267],{"className":1535,"style":196},[195],[138,1537,1540],{"transform":1538,"dataTid":1539},"translate(90.140,8.228)","pXsuBVQM01DMq13gbLIiD0g",[138,1541,1544,1547],{"dataTid":1542,"className":1543,"fill":175,"transform":176},"gXsuBVQM01DMq13gbLIiD0g",[174],[178,1545],{"x":180,"y":180,"href":1546},"#gl3Hi1Qo",[138,1548,1549],{"transform":184},[186,1550,1551],{"x":180,"y":188,"width":1517,"height":190},[192,1552,472],{"className":1553,"style":196},[195],[138,1555,1558],{"transform":1556,"dataTid":1557},"translate(105.419,8.228)","p7b\u002FDcebbJ0kdH\u002Fhkn7IqlI",[138,1559,1562,1565],{"dataTid":1560,"className":1561,"fill":175,"transform":176},"g7b\u002FDcebbJ0kdH\u002Fhkn7IqlI",[174],[178,1563],{"x":180,"y":180,"href":1564},"#gl3Hi1QIL",[138,1566,1567],{"transform":184},[186,1568,1570],{"x":180,"y":188,"width":1569,"height":190},"49.25",[192,1571,1573],{"className":1572,"style":196},[195],"𝑈",[138,1575,1577],{"transform":1576,"dataTid":1442},"translate(117.143,8.228)",[138,1578,1580,1582],{"dataTid":1445,"className":1579,"fill":175,"transform":176},[174],[178,1581],{"x":180,"y":180,"href":1449},[138,1583,1584],{"transform":184},[186,1585,1586],{"x":180,"y":188,"width":1454,"height":190},[192,1587,946],{"className":1588,"style":196},[195],[138,1590,1592],{"transform":1591,"dataTid":1251},"translate(128.756,8.228)",[138,1593,1595,1597],{"dataTid":1254,"className":1594,"fill":175,"transform":176},[174],[178,1596],{"x":180,"y":180,"href":1258},[138,1598,1599],{"transform":184},[186,1600,1601],{"x":180,"y":188,"width":1263,"height":190},[192,1602,1267],{"className":1603,"style":196},[195],[138,1605,1608],{"transform":1606,"dataTid":1607},"translate(140.870,8.228)","pzUvlOB5pueI9GCCk24jGCA",[138,1609,1612,1615],{"dataTid":1610,"className":1611,"fill":175,"transform":176},"gzUvlOB5pueI9GCCk24jGCA",[174],[178,1613],{"x":180,"y":180,"href":1614},"#gl3Hi1Qw",[138,1616,1617],{"transform":184},[186,1618,1619],{"x":180,"y":188,"width":1454,"height":190},[192,1620,298],{"className":1621,"style":196},[195],[138,1623,1626],{"transform":1624,"dataTid":1625},"translate(151.872,8.228)","pW80E1qSKMG0IwESBhXT5WM",[138,1627,1630,1633,1637,1640,1644,1648,1651,1655,1658,1661],{"dataTid":1628,"className":1629,"fill":175,"transform":176},"gW80E1qSKMG0IwESBhXT5WM",[174],[178,1631],{"x":180,"y":180,"href":1632},"#gl3Hi1SI",[178,1634],{"x":1635,"y":180,"href":1636},"750","#gl3Hi1VU",[178,1638],{"x":1639,"y":180,"href":1636},"1139",[178,1641],{"x":1642,"y":180,"href":1643},"1528","#gl3Hi1UY",[178,1645],{"x":1646,"y":180,"href":1647},"1972","#gl3Hi1U8",[178,1649],{"x":1650,"y":180,"href":1636},"2528",[178,1652],{"x":1653,"y":180,"href":1654},"2917","#gl3Hi1Uo",[178,1656],{"x":1657,"y":180,"href":1484},"3195",[178,1659],{"x":1660,"y":180,"href":1647},"3695",[138,1662,1663],{"transform":184},[186,1664,1666],{"x":180,"y":188,"width":1665,"height":190},"265.69",[192,1667,1669],{"className":1668,"style":196},[195],"Attention",[138,1671,1673],{"transform":1672,"dataTid":1505},"translate(198.633,8.228)",[138,1674,1676,1678],{"dataTid":1508,"className":1675,"fill":175,"transform":176},[174],[178,1677],{"x":180,"y":180,"href":1512},[138,1679,1680],{"transform":184},[186,1681,1682],{"x":180,"y":188,"width":1517,"height":190},[192,1683,238],{"className":1684,"style":196},[195],[138,1686,1688],{"transform":1687,"dataTid":1423},"translate(202.912,8.228)",[138,1689,1691,1693],{"dataTid":1426,"className":1690,"fill":175,"transform":176},[174],[178,1692],{"x":180,"y":180,"href":1430},[138,1694,1695],{"transform":184},[186,1696,1697],{"x":180,"y":188,"width":784,"height":190},[192,1698,1438],{"className":1699,"style":196},[195],[138,1701,1703],{"transform":1702,"dataTid":1539},"translate(212.911,8.228)",[138,1704,1706,1708],{"dataTid":1542,"className":1705,"fill":175,"transform":176},[174],[178,1707],{"x":180,"y":180,"href":1546},[138,1709,1710],{"transform":184},[186,1711,1712],{"x":180,"y":188,"width":1517,"height":190},[192,1713,472],{"className":1714,"style":196},[195],[138,1716,1717],{"transform":153,"dataTid":475},[138,1718],{"dataTid":478,"className":1719,"dataHint":15},[480],[138,1721,1722],{"transform":153,"dataTid":475},[138,1723],{"dataTid":478,"className":1724,"dataHint":15},[480],[51,1726,1731,1733,1792,1795,1797],{"style":53,"className":1727,"viewBox":1728,"width":1729,"height":1290,"dataWidth":1730,"dataHeight":1292,"xmlns":61,"xmlnsXLink":62,"xmlns:h5":63},[55],"0 0 190.000 11.000","17.272727272727273em","190.000",[65,1732,68],{"type":67},[70,1734,1736,1739,1742,1745,1748,1753,1756,1759,1764,1767,1770,1773,1776,1779,1782,1787],{"className":1735},[73],[75,1737],{"id":1300,"className":1738,"d":1302},[79],[75,1740],{"id":1305,"className":1741,"d":1307},[79],[75,1743],{"id":1310,"className":1744,"d":1312},[79],[75,1746],{"id":1315,"className":1747,"d":1317},[79],[75,1749],{"id":1750,"className":1751,"d":1752},"gl3Hi1S0",[79],"M 230 74 L 230 601 C 230 614 231 622 233 627 C 238 638 267 644 320 644 L 356 644 L 356 683 C 330 681 274 680 187 680 C 110 680 59 681 33 683 L 33 644 L 61 644 C 96 644 117 641 124 636 C 131 631 135 620 135 602 L 135 81 C 135 63 131 51 124 46 C 117 41 96 39 61 39 L 33 39 L 33 0 L 554 0 L 582 263 L 550 263 C 545 212 536 171 525 139 C 503 76 444 39 355 39 L 274 39 C 236 39 230 39 230 74 Z ",[75,1754],{"id":1325,"className":1755,"d":1327},[79],[75,1757],{"id":1330,"className":1758,"d":1332},[79],[75,1760],{"id":1761,"className":1762,"d":1763},"gl3Hi1TE",[79],"M 395 312 C 452 312 503 328 548 360 C 599 396 624 441 624 495 C 624 552 598 599 546 635 C 500 667 447 683 387 683 L 35 683 L 35 644 L 63 644 C 98 644 119 641 126 636 C 133 631 137 620 137 602 L 137 81 C 137 63 133 51 126 46 C 119 41 98 39 63 39 L 35 39 L 35 0 C 61 2 111 3 184 3 C 258 3 308 2 334 0 L 334 39 L 306 39 C 271 39 250 41 243 46 C 236 51 232 63 232 81 L 232 312 Z M 274 644 L 362 644 C 391 644 416 640 437 633 C 494 612 514 570 514 495 C 514 467 511 443 504 423 C 487 373 433 346 362 346 L 229 346 L 229 609 C 229 645 235 644 274 644 Z ",[75,1765],{"id":1335,"className":1766,"d":1337},[79],[75,1768],{"id":1340,"className":1769,"d":1342},[79],[75,1771],{"id":1355,"className":1772,"d":1357},[79],[75,1774],{"id":1365,"className":1775,"d":1367},[79],[75,1777],{"id":1370,"className":1778,"d":1372},[79],[75,1780],{"id":1385,"className":1781,"d":1387},[79],[75,1783],{"id":1784,"className":1785,"d":1786},"gl3Hi1QYL",[79],"M 664 680 C 646 680 584 683 565 683 C 550 683 542 675 542 660 C 542 649 548 644 561 644 C 582 644 593 638 593 626 C 593 615 584 600 567 581 L 342 322 L 236 609 C 233 617 231 622 230 625 C 230 638 248 644 284 644 C 303 644 312 652 312 668 C 312 678 306 683 293 683 L 168 680 C 146 680 76 683 58 683 C 43 683 35 675 35 659 C 35 649 44 644 62 644 C 80 644 95 642 104 640 C 123 635 124 632 130 613 L 252 288 C 255 281 256 276 256 273 L 235 190 C 223 139 215 107 211 94 C 204 67 197 51 190 46 C 183 41 160 39 122 39 C 100 38 91 37 91 15 C 91 5 97 0 109 0 L 234 3 L 360 0 C 375 -1 383 7 383 24 C 383 31 380 36 375 37 C 365 38 357 39 352 39 C 319 40 299 42 294 45 C 292 46 291 48 291 52 C 291 56 296 77 305 114 L 340 253 C 343 266 347 276 353 282 L 598 564 L 611 577 C 637 603 661 621 682 630 C 699 638 718 643 739 644 C 757 646 762 651 762 667 C 762 678 756 683 745 683 C 730 683 679 680 664 680 Z ",[75,1788],{"id":1789,"className":1790,"d":1791},"gl3Hi1QcL",[79],"M 720 653 C 722 662 723 668 723 670 C 723 679 718 683 707 683 L 277 683 C 252 683 249 680 243 660 L 189 483 C 187 478 186 472 185 467 C 188 459 190 453 201 453 C 206 453 212 457 218 465 C 239 530 267 577 302 604 C 337 631 389 644 457 644 L 609 644 L 62 33 L 58 13 C 58 4 64 0 75 0 L 516 0 C 542 0 545 2 551 23 L 621 242 C 622 246 625 253 625 257 C 625 267 619 272 608 272 C 601 272 594 263 588 244 C 566 176 539 126 506 94 C 471 59 413 42 333 42 L 173 42 Z ",[70,1793],{"className":1794},[134],[65,1796],{"type":67},[138,1798,1802],{"className":1799,"transform":142,"dataTid":1800,"dataPageWidth":1801,"dataPageHeight":1401},[141],"p7qT2oDFT1CoXu+qdpqNyo4","190",[138,1803,1806],{"dataTid":1804,"className":1805},"g7qT2oDFT1CoXu+qdpqNyo4",[150],[138,1807,1809],{"transform":153,"dataTid":1808},"pwUGbt8aAJKcmZc6Yn3TEqk",[138,1810,1813,2068],{"dataTid":1811,"className":1812},"gwUGbt8aAJKcmZc6Yn3TEqk",[150],[138,1814,1816],{"transform":153,"dataTid":1815},"pJNfLu5OUCNEFtaO0OIZwI4",[138,1817,1820,1839,1854,1881,1896,1911,1926,1946,1961,1976,1991,2018,2033,2048,2063],{"dataTid":1818,"className":1819},"gJNfLu5OUCNEFtaO0OIZwI4",[150],[138,1821,1823],{"transform":1422,"dataTid":1822},"pcTGOSxWiRSAZoCJjnE\u002Fc+k",[138,1824,1827,1830],{"dataTid":1825,"className":1826,"fill":175,"transform":176},"gcTGOSxWiRSAZoCJjnE\u002Fc+k",[174],[178,1828],{"x":180,"y":180,"href":1829},"#gl3Hi1QcL",[138,1831,1832],{"transform":184},[186,1833,1835],{"x":180,"y":188,"width":1834,"height":190},"46.94",[192,1836,1838],{"className":1837,"style":196},[195],"𝑍",[138,1840,1842],{"transform":1841,"dataTid":1442},"translate(11.317,8.228)",[138,1843,1845,1847],{"dataTid":1445,"className":1844,"fill":175,"transform":176},[174],[178,1846],{"x":180,"y":180,"href":1449},[138,1848,1849],{"transform":184},[186,1850,1851],{"x":180,"y":188,"width":1454,"height":190},[192,1852,946],{"className":1853,"style":196},[195],[138,1855,1857],{"transform":1856,"dataTid":1461},"translate(22.930,8.228)",[138,1858,1860,1862,1864,1866,1868,1870,1872,1874],{"dataTid":1464,"className":1859,"fill":175,"transform":176},[174],[178,1861],{"x":180,"y":180,"href":1468},[178,1863],{"x":1471,"y":180,"href":1472},[178,1865],{"x":1475,"y":180,"href":1476},[178,1867],{"x":1479,"y":180,"href":1480},[178,1869],{"x":1483,"y":180,"href":1484},[178,1871],{"x":1487,"y":180,"href":1488},[178,1873],{"x":1491,"y":180,"href":1492},[138,1875,1876],{"transform":184},[186,1877,1878],{"x":180,"y":188,"width":1497,"height":190},[192,1879,1501],{"className":1880,"style":196},[195],[138,1882,1884],{"transform":1883,"dataTid":1505},"translate(74.454,8.228)",[138,1885,1887,1889],{"dataTid":1508,"className":1886,"fill":175,"transform":176},[174],[178,1888],{"x":180,"y":180,"href":1512},[138,1890,1891],{"transform":184},[186,1892,1893],{"x":180,"y":188,"width":1517,"height":190},[192,1894,238],{"className":1895,"style":196},[195],[138,1897,1899],{"transform":1898,"dataTid":1557},"translate(78.733,8.228)",[138,1900,1902,1904],{"dataTid":1560,"className":1901,"fill":175,"transform":176},[174],[178,1903],{"x":180,"y":180,"href":1564},[138,1905,1906],{"transform":184},[186,1907,1908],{"x":180,"y":188,"width":1569,"height":190},[192,1909,1573],{"className":1910,"style":196},[195],[138,1912,1914],{"transform":1913,"dataTid":1539},"translate(87.401,8.228)",[138,1915,1917,1919],{"dataTid":1542,"className":1916,"fill":175,"transform":176},[174],[178,1918],{"x":180,"y":180,"href":1546},[138,1920,1921],{"transform":184},[186,1922,1923],{"x":180,"y":188,"width":1517,"height":190},[192,1924,472],{"className":1925,"style":196},[195],[138,1927,1930],{"transform":1928,"dataTid":1929},"translate(102.680,8.228)","pPx0QjCzB4xk7dZtT81ptME",[138,1931,1934,1937],{"dataTid":1932,"className":1933,"fill":175,"transform":176},"gPx0QjCzB4xk7dZtT81ptME",[174],[178,1935],{"x":180,"y":180,"href":1936},"#gl3Hi1QYL",[138,1938,1939],{"transform":184},[186,1940,1942],{"x":180,"y":188,"width":1941,"height":190},"49.38",[192,1943,1945],{"className":1944,"style":196},[195],"𝑌",[138,1947,1949],{"transform":1948,"dataTid":1442},"translate(114.426,8.228)",[138,1950,1952,1954],{"dataTid":1445,"className":1951,"fill":175,"transform":176},[174],[178,1953],{"x":180,"y":180,"href":1449},[138,1955,1956],{"transform":184},[186,1957,1958],{"x":180,"y":188,"width":1454,"height":190},[192,1959,946],{"className":1960,"style":196},[195],[138,1962,1964],{"transform":1963,"dataTid":1557},"translate(126.039,8.228)",[138,1965,1967,1969],{"dataTid":1560,"className":1966,"fill":175,"transform":176},[174],[178,1968],{"x":180,"y":180,"href":1564},[138,1970,1971],{"transform":184},[186,1972,1973],{"x":180,"y":188,"width":1569,"height":190},[192,1974,1573],{"className":1975,"style":196},[195],[138,1977,1979],{"transform":1978,"dataTid":1607},"translate(137.152,8.228)",[138,1980,1982,1984],{"dataTid":1610,"className":1981,"fill":175,"transform":176},[174],[178,1983],{"x":180,"y":180,"href":1614},[138,1985,1986],{"transform":184},[186,1987,1988],{"x":180,"y":188,"width":1454,"height":190},[192,1989,298],{"className":1990,"style":196},[195],[138,1992,1995],{"transform":1993,"dataTid":1994},"translate(148.154,8.228)","pYBYABJJp1BosFo6zLr87uI",[138,1996,1999,2001,2005,2009],{"dataTid":1997,"className":1998,"fill":175,"transform":176},"gYBYABJJp1BosFo6zLr87uI",[174],[178,2000],{"x":180,"y":180,"href":1472},[178,2002],{"x":2003,"y":180,"href":2004},"917","#gl3Hi1S0",[178,2006],{"x":2007,"y":180,"href":2008},"1542","#gl3Hi1TE",[138,2010,2011],{"transform":184},[186,2012,2014],{"x":180,"y":188,"width":2013,"height":190},"138.94",[192,2015,2017],{"className":2016,"style":196},[195],"MLP",[138,2019,2021],{"transform":2020,"dataTid":1505},"translate(172.607,8.228)",[138,2022,2024,2026],{"dataTid":1508,"className":2023,"fill":175,"transform":176},[174],[178,2025],{"x":180,"y":180,"href":1512},[138,2027,2028],{"transform":184},[186,2029,2030],{"x":180,"y":188,"width":1517,"height":190},[192,2031,238],{"className":2032,"style":196},[195],[138,2034,2036],{"transform":2035,"dataTid":1822},"translate(176.886,8.228)",[138,2037,2039,2041],{"dataTid":1825,"className":2038,"fill":175,"transform":176},[174],[178,2040],{"x":180,"y":180,"href":1829},[138,2042,2043],{"transform":184},[186,2044,2045],{"x":180,"y":188,"width":1834,"height":190},[192,2046,1838],{"className":2047,"style":196},[195],[138,2049,2051],{"transform":2050,"dataTid":1539},"translate(185.147,8.228)",[138,2052,2054,2056],{"dataTid":1542,"className":2053,"fill":175,"transform":176},[174],[178,2055],{"x":180,"y":180,"href":1546},[138,2057,2058],{"transform":184},[186,2059,2060],{"x":180,"y":188,"width":1517,"height":190},[192,2061,472],{"className":2062,"style":196},[195],[138,2064,2065],{"transform":153,"dataTid":475},[138,2066],{"dataTid":478,"className":2067,"dataHint":15},[480],[138,2069,2070],{"transform":153,"dataTid":475},[138,2071],{"dataTid":478,"className":2072,"dataHint":15},[480],[11,2074,2075],{},[850,2076],{"alt":2077,"src":2078},"A Transformer block with its two residual paths, one around attention and one around the MLP","\u002Fassets\u002Fblog\u002Ffrom-safetensors-to-tokens-executing-a-transformer-step-by-step\u002Ftransformer-block.svg",[11,2080,2081],{},"Foundry's decoder layer follows these four lines almost word for word.",[898,2083,2085],{"className":900,"code":2084,"language":902,"meta":903,"style":903},"let normed = self.input_layernorm.forward(device, hidden_states)?;\nlet attn_output = self.self_attn.forward(device, &normed, position_embeddings, attention_mask, past_key_values, past_seen_tokens)?;\nlet hidden_states = hidden_states.add(device, &attn_output)?;\nlet normed = self.post_attention_layernorm.forward(device, &hidden_states)?;\nlet mlp_output = self.mlp.forward(device, &normed)?;\nhidden_states.add(device, &mlp_output)\n",[868,2086,2087,2115,2146,2174,2204,2235],{"__ignoreMap":903},[907,2088,2089,2091,2094,2096,2098,2100,2103,2105,2108,2111,2113],{"class":909,"line":910},[907,2090,940],{"class":913},[907,2092,2093],{"class":926}," normed ",[907,2095,946],{"class":913},[907,2097,1177],{"class":1007},[907,2099,1072],{"class":913},[907,2101,2102],{"class":926},"input_layernorm",[907,2104,1072],{"class":913},[907,2106,2107],{"class":916},"forward",[907,2109,2110],{"class":926},"(device, hidden_states)",[907,2112,965],{"class":913},[907,2114,927],{"class":926},[907,2116,2117,2119,2122,2124,2126,2128,2131,2133,2135,2137,2139,2142,2144],{"class":909,"line":930},[907,2118,940],{"class":913},[907,2120,2121],{"class":926}," attn_output ",[907,2123,946],{"class":913},[907,2125,1177],{"class":1007},[907,2127,1072],{"class":913},[907,2129,2130],{"class":926},"self_attn",[907,2132,1072],{"class":913},[907,2134,2107],{"class":916},[907,2136,1190],{"class":926},[907,2138,1198],{"class":913},[907,2140,2141],{"class":926},"normed, position_embeddings, attention_mask, past_key_values, past_seen_tokens)",[907,2143,965],{"class":913},[907,2145,927],{"class":926},[907,2147,2148,2150,2153,2155,2158,2160,2163,2165,2167,2170,2172],{"class":909,"line":937},[907,2149,940],{"class":913},[907,2151,2152],{"class":926}," hidden_states ",[907,2154,946],{"class":913},[907,2156,2157],{"class":926}," hidden_states",[907,2159,1072],{"class":913},[907,2161,2162],{"class":916},"add",[907,2164,1190],{"class":926},[907,2166,1198],{"class":913},[907,2168,2169],{"class":926},"attn_output)",[907,2171,965],{"class":913},[907,2173,927],{"class":926},[907,2175,2176,2178,2180,2182,2184,2186,2189,2191,2193,2195,2197,2200,2202],{"class":909,"line":970},[907,2177,940],{"class":913},[907,2179,2093],{"class":926},[907,2181,946],{"class":913},[907,2183,1177],{"class":1007},[907,2185,1072],{"class":913},[907,2187,2188],{"class":926},"post_attention_layernorm",[907,2190,1072],{"class":913},[907,2192,2107],{"class":916},[907,2194,1190],{"class":926},[907,2196,1198],{"class":913},[907,2198,2199],{"class":926},"hidden_states)",[907,2201,965],{"class":913},[907,2203,927],{"class":926},[907,2205,2206,2208,2211,2213,2215,2217,2220,2222,2224,2226,2228,2231,2233],{"class":909,"line":985},[907,2207,940],{"class":913},[907,2209,2210],{"class":926}," mlp_output ",[907,2212,946],{"class":913},[907,2214,1177],{"class":1007},[907,2216,1072],{"class":913},[907,2218,2219],{"class":926},"mlp",[907,2221,1072],{"class":913},[907,2223,2107],{"class":916},[907,2225,1190],{"class":926},[907,2227,1198],{"class":913},[907,2229,2230],{"class":926},"normed)",[907,2232,965],{"class":913},[907,2234,927],{"class":926},[907,2236,2237,2240,2242,2244,2246,2248],{"class":909,"line":1001},[907,2238,2239],{"class":926},"hidden_states",[907,2241,1072],{"class":913},[907,2243,2162],{"class":916},[907,2245,1190],{"class":926},[907,2247,1198],{"class":913},[907,2249,2250],{"class":926},"mlp_output)\n",[11,2252,2253],{},"RMSNorm adjusts the scale of each vector. It squares the values, takes their mean, adds a small constant and takes the square root. It divides the original values by that result and applies a learned scale. Foundry does the sums in F32, a 32-bit format, to reduce rounding errors. It converts the normalized result back to BF16 before applying the learned scale.",[898,2255,2257],{"className":900,"code":2256,"language":902,"meta":903,"style":903},"let x_f32 = x.to_dtype(device, DType::F32)?;\nlet scale = x_f32\n    .mul(device, &x_f32)?\n    .sum_keepdim(device, dim)?\n    .mul(device, &self.dim_inv)?\n    .add(device, &self.eps)?\n    .sqrt(device)?\n    .reciprocal(device)?;\nx_f32.mul(device, &scale)?.to_dtype(device, x.dtype())?.mul(device, &self.weight)\n",[868,2258,2259,2292,2304,2322,2334,2354,2373,2385,2398],{"__ignoreMap":903},[907,2260,2261,2263,2266,2268,2271,2273,2276,2278,2281,2283,2286,2288,2290],{"class":909,"line":910},[907,2262,940],{"class":913},[907,2264,2265],{"class":926}," x_f32 ",[907,2267,946],{"class":913},[907,2269,2270],{"class":926}," x",[907,2272,1072],{"class":913},[907,2274,2275],{"class":916},"to_dtype",[907,2277,1190],{"class":926},[907,2279,2280],{"class":916},"DType",[907,2282,920],{"class":913},[907,2284,2285],{"class":916},"F32",[907,2287,472],{"class":926},[907,2289,965],{"class":913},[907,2291,927],{"class":926},[907,2293,2294,2296,2299,2301],{"class":909,"line":930},[907,2295,940],{"class":913},[907,2297,2298],{"class":926}," scale ",[907,2300,946],{"class":913},[907,2302,2303],{"class":926}," x_f32\n",[907,2305,2306,2309,2312,2314,2316,2319],{"class":909,"line":937},[907,2307,2308],{"class":913},"    .",[907,2310,2311],{"class":916},"mul",[907,2313,1190],{"class":926},[907,2315,1198],{"class":913},[907,2317,2318],{"class":926},"x_f32)",[907,2320,2321],{"class":913},"?\n",[907,2323,2324,2326,2329,2332],{"class":909,"line":970},[907,2325,2308],{"class":913},[907,2327,2328],{"class":916},"sum_keepdim",[907,2330,2331],{"class":926},"(device, dim)",[907,2333,2321],{"class":913},[907,2335,2336,2338,2340,2342,2344,2347,2349,2352],{"class":909,"line":985},[907,2337,2308],{"class":913},[907,2339,2311],{"class":916},[907,2341,1190],{"class":926},[907,2343,1198],{"class":913},[907,2345,2346],{"class":1007},"self",[907,2348,1072],{"class":913},[907,2350,2351],{"class":926},"dim_inv)",[907,2353,2321],{"class":913},[907,2355,2356,2358,2360,2362,2364,2366,2368,2371],{"class":909,"line":1001},[907,2357,2308],{"class":913},[907,2359,2162],{"class":916},[907,2361,1190],{"class":926},[907,2363,1198],{"class":913},[907,2365,2346],{"class":1007},[907,2367,1072],{"class":913},[907,2369,2370],{"class":926},"eps)",[907,2372,2321],{"class":913},[907,2374,2375,2377,2380,2383],{"class":909,"line":1014},[907,2376,2308],{"class":913},[907,2378,2379],{"class":916},"sqrt",[907,2381,2382],{"class":926},"(device)",[907,2384,2321],{"class":913},[907,2386,2387,2389,2392,2394,2396],{"class":909,"line":1031},[907,2388,2308],{"class":913},[907,2390,2391],{"class":916},"reciprocal",[907,2393,2382],{"class":926},[907,2395,965],{"class":913},[907,2397,927],{"class":926},[907,2399,2400,2403,2405,2407,2409,2411,2414,2417,2419,2422,2424,2427,2430,2432,2434,2436,2438,2440,2442],{"class":909,"line":1042},[907,2401,2402],{"class":926},"x_f32",[907,2404,1072],{"class":913},[907,2406,2311],{"class":916},[907,2408,1190],{"class":926},[907,2410,1198],{"class":913},[907,2412,2413],{"class":926},"scale)",[907,2415,2416],{"class":913},"?.",[907,2418,2275],{"class":916},[907,2420,2421],{"class":926},"(device, x",[907,2423,1072],{"class":913},[907,2425,2426],{"class":916},"dtype",[907,2428,2429],{"class":926},"())",[907,2431,2416],{"class":913},[907,2433,2311],{"class":916},[907,2435,1190],{"class":926},[907,2437,1198],{"class":913},[907,2439,2346],{"class":1007},[907,2441,1072],{"class":913},[907,2443,2444],{"class":926},"weight)\n",[43,2446,2448],{"id":2447},"queries-keys-and-values","Queries, keys and values",[11,2450,2451],{},"Attention starts with three matrix multiplications. The normalized vectors are multiplied by three weight matrices from the file.",[51,2453,2460,2462,2506,2509,2511],{"style":53,"className":2454,"viewBox":2455,"width":2456,"height":2457,"dataWidth":2458,"dataHeight":2459,"xmlns":61,"xmlnsXLink":62,"xmlns:h5":63},[55],"0 0 175.000 12.000","15.909090909090908em","1.0909090909090908em","175.000","12.000",[65,2461,68],{"type":67},[70,2463,2465,2468,2471,2476,2481,2486,2491,2496,2501],{"className":2464},[73],[75,2466],{"id":1315,"className":2467,"d":1317},[79],[75,2469],{"id":1380,"className":2470,"d":1382},[79],[75,2472],{"id":2473,"className":2474,"d":2475},"gl3Hi1fgK",[79],"M 804 680 C 783 680 715 683 694 683 C 679 683 671 675 671 659 C 671 651 678 646 692 643 C 706 640 714 635 714 628 C 714 617 703 602 680 585 L 302 296 L 379 602 C 384 623 393 636 406 641 C 412 643 429 644 458 644 C 484 644 496 644 496 668 C 496 678 490 683 478 683 L 351 680 L 223 683 C 207 684 199 676 199 660 C 199 649 210 644 232 644 C 252 644 293 647 293 631 C 293 629 292 623 289 613 L 156 82 C 151 60 141 47 126 42 C 119 40 101 39 70 39 C 48 39 38 37 38 16 C 38 5 44 0 57 0 L 183 3 L 246 2 C 257 2 299 0 310 0 C 326 0 334 8 334 24 C 334 34 324 39 303 39 C 262 39 242 43 242 52 C 242 57 245 72 251 95 L 290 249 L 433 359 L 546 98 C 553 81 557 70 557 65 C 557 48 541 39 510 39 C 492 39 484 35 484 15 C 484 5 490 0 502 0 C 524 0 600 3 622 3 C 641 3 701 0 720 0 C 734 0 741 8 741 24 C 741 33 733 38 718 39 C 677 39 667 50 653 81 L 512 404 C 511 409 509 413 508 416 L 708 570 C 757 608 799 631 833 639 C 857 643 889 642 889 668 C 889 678 884 683 874 683 C 860 683 818 680 804 680 Z ",[75,2477],{"id":2478,"className":2479,"d":2480},"gl3Hi1f4K",[79],"M 306 -24 C 335 -24 366 -19 399 -10 C 396 -48 395 -72 395 -81 C 395 -156 424 -194 481 -194 C 542 -194 589 -145 612 -100 C 637 -48 650 -14 650 2 C 650 11 645 16 635 16 C 628 16 623 12 620 3 C 605 -43 564 -90 508 -90 C 457 -90 450 -49 442 6 C 526 39 597 96 654 178 C 711 260 740 345 740 434 C 740 513 717 577 670 628 C 623 679 561 705 484 705 C 431 705 378 692 325 665 C 217 612 138 528 87 415 C 62 358 49 302 49 245 C 49 88 151 -24 306 -24 Z M 645 466 C 645 405 630 335 600 254 C 565 161 511 92 436 48 C 432 70 426 90 418 108 C 405 138 381 153 346 153 C 291 153 241 103 241 48 C 241 39 242 30 244 23 C 177 54 144 117 144 213 C 144 255 152 306 169 366 C 204 493 267 583 358 636 C 401 660 441 672 478 672 C 588 672 645 581 645 466 Z M 346 123 C 383 123 402 96 402 43 C 402 34 399 29 393 27 C 365 15 338 9 312 9 C 284 9 270 22 270 48 C 270 86 308 123 346 123 Z ",[75,2482],{"id":2483,"className":2484,"d":2485},"gl3Hi1QML",[79],"M 671 680 C 652 680 592 683 573 683 C 558 683 550 675 550 660 C 550 650 557 645 570 644 C 598 643 612 633 612 616 C 612 607 607 595 598 580 L 300 107 L 234 619 C 234 636 255 644 298 644 C 318 644 327 651 327 668 C 327 678 321 683 309 683 C 287 683 209 680 187 680 C 167 680 99 683 79 683 C 64 683 56 675 56 660 C 56 649 66 644 85 644 C 102 644 114 642 122 640 C 138 635 137 633 140 614 L 218 4 C 221 -13 229 -22 242 -22 C 255 -22 265 -15 273 -2 L 629 564 C 652 600 674 623 696 632 C 711 639 730 643 752 644 C 763 645 768 652 769 667 C 770 678 764 683 752 683 C 737 683 686 680 671 680 Z ",[75,2487],{"id":2488,"className":2489,"d":2490},"gl3Hi1QQL",[79],"M 956 680 C 937 680 872 683 853 683 C 838 683 830 675 830 660 C 830 650 836 645 848 644 C 885 643 903 632 903 610 C 903 604 899 594 891 579 L 629 125 L 594 602 C 590 638 614 644 659 644 C 680 644 690 652 690 668 C 690 678 684 683 671 683 C 649 683 572 680 550 680 C 531 680 465 683 446 683 C 431 683 423 675 423 659 C 423 649 432 644 451 644 C 482 644 498 637 499 623 L 505 548 L 261 125 L 225 613 C 222 638 264 644 296 644 C 313 644 321 652 321 668 C 321 678 315 683 303 683 C 281 683 203 680 181 680 C 162 680 96 683 77 683 C 62 683 55 675 55 660 C 55 649 65 644 84 644 C 126 644 130 642 132 612 L 177 7 C 178 -12 185 -22 197 -22 C 210 -20 215 -17 223 -2 C 224 -1 224 0 224 1 L 509 494 L 545 7 C 546 -12 553 -22 565 -22 C 575 -22 584 -15 592 -1 L 918 562 C 945 611 964 639 1025 644 C 1043 646 1048 651 1048 668 C 1048 678 1043 683 1032 683 C 1018 683 970 680 956 680 Z ",[75,2492],{"id":2493,"className":2494,"d":2495},"gl3Hi1bwT",[79],"M 894 679 C 872 679 796 683 774 683 C 757 683 749 674 749 657 C 752 647 753 641 768 640 C 782 639 789 635 789 630 C 789 619 775 604 747 584 L 342 300 L 418 601 C 423 621 433 633 448 638 C 453 639 470 640 501 640 C 524 640 535 645 535 664 C 535 677 528 683 515 683 C 490 683 410 679 385 679 C 338 679 276 683 254 683 C 238 683 230 674 230 657 C 230 646 242 640 266 640 C 301 640 322 642 322 628 L 186 82 C 181 62 171 50 156 46 C 151 44 133 43 103 43 C 80 43 69 40 69 17 C 72 6 73 0 89 0 C 113 0 194 4 218 4 C 265 4 328 0 350 0 C 366 0 374 9 374 26 C 374 35 371 40 364 41 C 353 42 345 43 339 43 C 309 44 282 42 282 56 L 328 244 L 488 356 L 630 82 C 634 74 636 68 636 65 C 636 50 622 43 594 43 C 573 43 565 36 565 17 C 568 6 569 0 585 0 C 609 0 688 4 712 4 C 731 4 790 0 809 0 C 825 0 833 8 833 25 C 833 37 824 43 807 43 C 776 43 755 53 745 74 C 687 187 628 298 571 412 L 592 429 L 692 499 C 766 550 807 579 815 584 C 870 618 916 637 953 640 C 970 641 979 650 979 665 C 979 676 973 682 962 682 C 950 682 906 679 894 679 Z ",[75,2497],{"id":2498,"className":2499,"d":2500},"gl3Hi1cIT",[79],"M 361 -22 C 392 -22 426 -18 461 -9 L 461 -57 C 461 -94 467 -124 478 -149 C 491 -179 518 -194 559 -194 C 621 -194 671 -147 696 -103 C 725 -53 740 -18 740 2 C 740 14 734 20 722 20 C 713 20 707 15 705 5 C 690 -41 645 -82 586 -82 C 535 -82 523 -44 512 8 C 599 39 673 93 734 169 C 797 249 829 334 829 423 C 829 507 801 575 746 626 C 691 677 622 703 538 703 C 460 703 385 681 312 638 C 193 567 69 426 69 254 C 69 88 192 -22 361 -22 Z M 725 450 C 725 391 710 324 679 249 C 644 160 585 95 502 54 C 483 122 449 156 400 156 C 344 156 286 109 286 53 C 286 46 286 39 287 34 C 211 64 173 129 173 229 C 173 342 210 444 283 534 C 335 598 426 663 530 663 C 652 663 725 575 725 450 Z M 400 120 C 438 120 458 92 461 35 C 430 24 399 18 368 18 C 337 18 321 30 321 53 C 321 91 362 120 400 120 Z ",[75,2502],{"id":2503,"className":2504,"d":2505},"gl3Hi1ccT",[79],"M 764 679 C 745 679 680 683 661 683 C 645 683 637 674 637 657 C 637 646 644 640 659 639 C 684 638 696 630 696 615 C 696 607 690 594 679 577 L 359 107 L 271 603 C 270 610 269 614 269 617 C 269 632 288 639 327 639 C 348 639 361 644 361 665 L 361 669 C 359 678 352 683 340 683 C 317 683 236 679 213 679 C 192 679 124 683 103 683 C 87 683 79 674 79 657 C 79 645 89 639 108 639 C 151 639 160 642 165 611 L 273 10 C 276 -10 286 -20 301 -20 C 315 -20 327 -12 338 3 L 716 559 C 750 609 790 636 835 640 C 853 641 861 648 861 665 C 861 677 855 683 843 683 C 829 683 778 679 764 679 Z ",[70,2507],{"className":2508},[134],[65,2510],{"type":67},[138,2512,2517],{"className":2513,"transform":142,"dataTid":2514,"dataPageWidth":2515,"dataPageHeight":2516},[141],"pUKXWS\u002FxQmDkI5p33I7uRB8","175","12",[138,2518,2521],{"dataTid":2519,"className":2520},"gUKXWS\u002FxQmDkI5p33I7uRB8",[150],[138,2522,2524],{"transform":153,"dataTid":2523},"pXcCsu2pTFgoy7NgQQSxTY4",[138,2525,2528,2797],{"dataTid":2526,"className":2527},"gXcCsu2pTFgoy7NgQQSxTY4",[150],[138,2529,2531],{"transform":153,"dataTid":2530},"pPAJMglnTlSIl3TQXv5Vk5Y",[138,2532,2535,2555,2570,2585,2605,2624,2644,2659,2674,2689,2708,2728,2743,2758,2773,2792],{"dataTid":2533,"className":2534},"gPAJMglnTlSIl3TQXv5Vk5Y",[150],[138,2536,2539],{"transform":2537,"dataTid":2538},"translate(0.000,7.755)","pfIcyfb5\u002FQuodN0tOoKBsKA",[138,2540,2543,2546],{"dataTid":2541,"className":2542,"fill":175,"transform":176},"gfIcyfb5\u002FQuodN0tOoKBsKA",[174],[178,2544],{"x":180,"y":180,"href":2545},"#gl3Hi1f4K",[138,2547,2548],{"transform":184},[186,2549,2551],{"x":180,"y":188,"width":2550,"height":190},"49.44",[192,2552,2554],{"className":2553,"style":196},[195],"𝑄",[138,2556,2558],{"transform":2557,"dataTid":1442},"translate(11.757,7.755)",[138,2559,2561,2563],{"dataTid":1445,"className":2560,"fill":175,"transform":176},[174],[178,2562],{"x":180,"y":180,"href":1449},[138,2564,2565],{"transform":184},[186,2566,2567],{"x":180,"y":188,"width":1454,"height":190},[192,2568,946],{"className":2569,"style":196},[195],[138,2571,2573],{"transform":2572,"dataTid":1423},"translate(23.370,7.755)",[138,2574,2576,2578],{"dataTid":1426,"className":2575,"fill":175,"transform":176},[174],[178,2577],{"x":180,"y":180,"href":1430},[138,2579,2580],{"transform":184},[186,2581,2582],{"x":180,"y":188,"width":784,"height":190},[192,2583,1438],{"className":2584,"style":196},[195],[138,2586,2589],{"transform":2587,"dataTid":2588},"translate(33.369,7.755)","p1J330TwXsYwqOESbbfpOj0",[138,2590,2593,2596],{"dataTid":2591,"className":2592,"fill":175,"transform":176},"g1J330TwXsYwqOESbbfpOj0",[174],[178,2594],{"x":180,"y":180,"href":2595},"#gl3Hi1QQL",[138,2597,2598],{"transform":184},[186,2599,2601],{"x":180,"y":188,"width":2600,"height":190},"67.25",[192,2602,2604],{"className":2603,"style":196},[195],"𝑊",[138,2606,2609],{"transform":2607,"dataTid":2608},"translate(42.653,10.472)","ptpz26kUxT2Ei4\u002F7Km0K+O4",[138,2610,2613,2616],{"dataTid":2611,"className":2612,"fill":175,"transform":206},"gtpz26kUxT2Ei4\u002F7Km0K+O4",[174],[178,2614],{"x":180,"y":180,"href":2615},"#gl3Hi1cIT",[138,2617,2618],{"transform":184},[186,2619,2621],{"x":180,"y":188,"width":2620,"height":190},"56.25",[192,2622,2554],{"className":2623,"style":196},[195],[138,2625,2628],{"transform":2626,"dataTid":2627},"translate(61.199,7.755)","pjXYRl9+L0EEolYug2DunfY",[138,2629,2632,2635],{"dataTid":2630,"className":2631,"fill":175,"transform":176},"gjXYRl9+L0EEolYug2DunfY",[174],[178,2633],{"x":180,"y":180,"href":2634},"#gl3Hi1fgK",[138,2636,2637],{"transform":184},[186,2638,2640],{"x":180,"y":188,"width":2639,"height":190},"57.31",[192,2641,2643],{"className":2642,"style":196},[195],"𝐾",[138,2645,2647],{"transform":2646,"dataTid":1442},"translate(74.342,7.755)",[138,2648,2650,2652],{"dataTid":1445,"className":2649,"fill":175,"transform":176},[174],[178,2651],{"x":180,"y":180,"href":1449},[138,2653,2654],{"transform":184},[186,2655,2656],{"x":180,"y":188,"width":1454,"height":190},[192,2657,946],{"className":2658,"style":196},[195],[138,2660,2662],{"transform":2661,"dataTid":1423},"translate(85.955,7.755)",[138,2663,2665,2667],{"dataTid":1426,"className":2664,"fill":175,"transform":176},[174],[178,2666],{"x":180,"y":180,"href":1430},[138,2668,2669],{"transform":184},[186,2670,2671],{"x":180,"y":188,"width":784,"height":190},[192,2672,1438],{"className":2673,"style":196},[195],[138,2675,2677],{"transform":2676,"dataTid":2588},"translate(95.954,7.755)",[138,2678,2680,2682],{"dataTid":2591,"className":2679,"fill":175,"transform":176},[174],[178,2681],{"x":180,"y":180,"href":2595},[138,2683,2684],{"transform":184},[186,2685,2686],{"x":180,"y":188,"width":2600,"height":190},[192,2687,2604],{"className":2688,"style":196},[195],[138,2690,2693],{"transform":2691,"dataTid":2692},"translate(105.238,10.472)","pe0jbh2OazrQSo4iGkvaRSY",[138,2694,2697,2700],{"dataTid":2695,"className":2696,"fill":175,"transform":206},"ge0jbh2OazrQSo4iGkvaRSY",[174],[178,2698],{"x":180,"y":180,"href":2699},"#gl3Hi1bwT",[138,2701,2702],{"transform":184},[186,2703,2705],{"x":180,"y":188,"width":2704,"height":190},"62.94",[192,2706,2643],{"className":2707,"style":196},[195],[138,2709,2712],{"transform":2710,"dataTid":2711},"translate(124.608,7.755)","p9ivEDomtzaU3Ug0oyx+0kI",[138,2713,2716,2719],{"dataTid":2714,"className":2715,"fill":175,"transform":176},"g9ivEDomtzaU3Ug0oyx+0kI",[174],[178,2717],{"x":180,"y":180,"href":2718},"#gl3Hi1QML",[138,2720,2721],{"transform":184},[186,2722,2724],{"x":180,"y":188,"width":2723,"height":190},"49.81",[192,2725,2727],{"className":2726,"style":196},[195],"𝑉",[138,2729,2731],{"transform":2730,"dataTid":1442},"translate(136.431,7.755)",[138,2732,2734,2736],{"dataTid":1445,"className":2733,"fill":175,"transform":176},[174],[178,2735],{"x":180,"y":180,"href":1449},[138,2737,2738],{"transform":184},[186,2739,2740],{"x":180,"y":188,"width":1454,"height":190},[192,2741,946],{"className":2742,"style":196},[195],[138,2744,2746],{"transform":2745,"dataTid":1423},"translate(148.044,7.755)",[138,2747,2749,2751],{"dataTid":1426,"className":2748,"fill":175,"transform":176},[174],[178,2750],{"x":180,"y":180,"href":1430},[138,2752,2753],{"transform":184},[186,2754,2755],{"x":180,"y":188,"width":784,"height":190},[192,2756,1438],{"className":2757,"style":196},[195],[138,2759,2761],{"transform":2760,"dataTid":2588},"translate(158.043,7.755)",[138,2762,2764,2766],{"dataTid":2591,"className":2763,"fill":175,"transform":176},[174],[178,2765],{"x":180,"y":180,"href":2595},[138,2767,2768],{"transform":184},[186,2769,2770],{"x":180,"y":188,"width":2600,"height":190},[192,2771,2604],{"className":2772,"style":196},[195],[138,2774,2777],{"transform":2775,"dataTid":2776},"translate(167.327,10.472)","pwONjGRyhSBQE6HwQZu3e88",[138,2778,2781,2784],{"dataTid":2779,"className":2780,"fill":175,"transform":206},"gwONjGRyhSBQE6HwQZu3e88",[174],[178,2782],{"x":180,"y":180,"href":2783},"#gl3Hi1ccT",[138,2785,2786],{"transform":184},[186,2787,2789],{"x":180,"y":188,"width":2788,"height":190},"55.56",[192,2790,2727],{"className":2791,"style":196},[195],[138,2793,2794],{"transform":153,"dataTid":475},[138,2795],{"dataTid":478,"className":2796,"dataHint":15},[480],[138,2798,2799],{"transform":153,"dataTid":475},[138,2800],{"dataTid":478,"className":2801,"dataHint":15},[480],[11,2803,2804],{},"Each query is compared with the keys to give an attention score for every position. The scores determine how much of each value to use. Queries, keys and values are split into groups called heads. Each head has 128 numbers and its own attention scores.",[11,2806,2807],{},[850,2808],{"alt":2809,"src":2810},"Queries with 24 heads, keys and values with 8 heads each, and how every key and value head serves 3 query heads","\u002Fassets\u002Fblog\u002Ffrom-safetensors-to-tokens-executing-a-transformer-step-by-step\u002Fqkv.svg",[11,2812,2813,2814,2816,2817,2820],{},"Llama 3.2 3B has 24 query heads but only 8 key heads and 8 value heads. So Q has shape ",[868,2815,1276],{}," and K and V have shape ",[868,2818,2819],{},"[1, 47, 1024]",". Each key and value head is shared by a group of 3 query heads. This is grouped-query attention. It makes keys and values three times smaller than they would be with one pair per query head.",[11,2822,2823,2824,2827],{},"Foundry applies each projection with a ",[868,2825,2826],{},"Linear"," operation, then reshapes the result into heads.",[898,2829,2831],{"className":900,"code":2830,"language":902,"meta":903,"style":903},"let query_states = self.q_proj.forward(device, hidden_states)?.reshape(query_shape)?;\nlet key_states = self.k_proj.forward(device, hidden_states)?.reshape(kv_shape)?;\nlet value_states = self.v_proj.forward(device, hidden_states)?.reshape(kv_shape)?.permute(&HEAD_MAJOR)?;\n",[868,2832,2833,2866,2899],{"__ignoreMap":903},[907,2834,2835,2837,2840,2842,2844,2846,2849,2851,2853,2855,2857,2859,2862,2864],{"class":909,"line":910},[907,2836,940],{"class":913},[907,2838,2839],{"class":926}," query_states ",[907,2841,946],{"class":913},[907,2843,1177],{"class":1007},[907,2845,1072],{"class":913},[907,2847,2848],{"class":926},"q_proj",[907,2850,1072],{"class":913},[907,2852,2107],{"class":916},[907,2854,2110],{"class":926},[907,2856,2416],{"class":913},[907,2858,1128],{"class":916},[907,2860,2861],{"class":926},"(query_shape)",[907,2863,965],{"class":913},[907,2865,927],{"class":926},[907,2867,2868,2870,2873,2875,2877,2879,2882,2884,2886,2888,2890,2892,2895,2897],{"class":909,"line":930},[907,2869,940],{"class":913},[907,2871,2872],{"class":926}," key_states ",[907,2874,946],{"class":913},[907,2876,1177],{"class":1007},[907,2878,1072],{"class":913},[907,2880,2881],{"class":926},"k_proj",[907,2883,1072],{"class":913},[907,2885,2107],{"class":916},[907,2887,2110],{"class":926},[907,2889,2416],{"class":913},[907,2891,1128],{"class":916},[907,2893,2894],{"class":926},"(kv_shape)",[907,2896,965],{"class":913},[907,2898,927],{"class":926},[907,2900,2901,2903,2906,2908,2910,2912,2915,2917,2919,2921,2923,2925,2927,2929,2932,2934,2936,2939,2941,2943],{"class":909,"line":937},[907,2902,940],{"class":913},[907,2904,2905],{"class":926}," value_states ",[907,2907,946],{"class":913},[907,2909,1177],{"class":1007},[907,2911,1072],{"class":913},[907,2913,2914],{"class":926},"v_proj",[907,2916,1072],{"class":913},[907,2918,2107],{"class":916},[907,2920,2110],{"class":926},[907,2922,2416],{"class":913},[907,2924,1128],{"class":916},[907,2926,2894],{"class":926},[907,2928,2416],{"class":913},[907,2930,2931],{"class":916},"permute",[907,2933,238],{"class":926},[907,2935,1198],{"class":913},[907,2937,2938],{"class":1007},"HEAD_MAJOR",[907,2940,472],{"class":926},[907,2942,965],{"class":913},[907,2944,927],{"class":926},[11,2946,2947],{},"Attention on its own doesn't know the order of the tokens. RoPE adds it. Each query and each key is cut into pairs of numbers, and each pair is rotated by an angle that grows with the position of the token. After the rotation, the match between a query and a key depends on how far apart the two tokens are. Values are not rotated.",[11,2949,2950],{},"Different pairs rotate at different rates. The model configuration sets these rates, using a base of 500,000 and Llama 3 position scaling. The code below computes a starting frequency. The scaling rule adjusts some of these frequencies for longer contexts.",[898,2952,2954],{"className":900,"code":2953,"language":902,"meta":903,"style":903},"let inv_frequency = theta\n    .powf(exact_f32(dim_index).map_err(too_large_for_f32)? \u002F head_dim_f32)\n    .recip();\n",[868,2955,2956,2968,2999],{"__ignoreMap":903},[907,2957,2958,2960,2963,2965],{"class":909,"line":910},[907,2959,940],{"class":913},[907,2961,2962],{"class":926}," inv_frequency ",[907,2964,946],{"class":913},[907,2966,2967],{"class":926}," theta\n",[907,2969,2970,2972,2975,2977,2980,2983,2985,2988,2991,2993,2996],{"class":909,"line":930},[907,2971,2308],{"class":913},[907,2973,2974],{"class":916},"powf",[907,2976,238],{"class":926},[907,2978,2979],{"class":916},"exact_f32",[907,2981,2982],{"class":926},"(dim_index)",[907,2984,1072],{"class":913},[907,2986,2987],{"class":916},"map_err",[907,2989,2990],{"class":926},"(too_large_for_f32)",[907,2992,965],{"class":913},[907,2994,2995],{"class":913}," \u002F",[907,2997,2998],{"class":926}," head_dim_f32)\n",[907,3000,3001,3003,3006],{"class":909,"line":937},[907,3002,2308],{"class":913},[907,3004,3005],{"class":916},"recip",[907,3007,3008],{"class":926},"();\n",[43,3010,3012],{"id":3011},"a-closer-look-at-softmax","A closer look at softmax",[11,3014,3015],{},"The heart of attention is three operations in a row. A matrix multiplication compares each query with each key. Softmax turns these comparisons into weights. A second matrix multiplication uses the weights to mix the values.",[51,3017,3024,3026,3116,3119,3121],{"style":53,"className":3018,"viewBox":3019,"width":3020,"height":3021,"dataWidth":3022,"dataHeight":3023,"xmlns":61,"xmlnsXLink":62,"xmlns:h5":63},[55],"0 0 185.000 33.000","16.818181818181817em","3em","185.000","33.000",[65,3025,68],{"type":67},[70,3027,3029,3032,3035,3040,3045,3048,3051,3056,3059,3064,3069,3072,3077,3082,3085,3088,3093,3096,3101,3106,3111],{"className":3028},[73],[75,3030],{"id":1310,"className":3031,"d":1312},[79],[75,3033],{"id":1315,"className":3034,"d":1317},[79],[75,3036],{"id":3037,"className":3038,"d":3039},"gl3Hi1UI",[79],"M 483 91 L 483 150 L 451 150 L 451 91 C 451 52 440 32 419 32 C 399 32 387 57 387 77 L 387 274 C 387 309 383 335 376 353 C 352 412 283 448 213 448 C 136 448 60 405 60 333 C 60 300 77 283 110 283 C 143 283 159 299 159 332 C 159 361 144 378 113 381 C 135 406 168 419 211 419 C 273 419 311 362 311 297 L 311 264 C 233 259 174 247 133 228 C 66 197 32 154 32 97 C 32 71 42 49 61 32 C 93 3 137 -11 193 -11 C 252 -11 294 14 320 65 C 327 27 355 -6 398 -6 C 451 -6 483 36 483 91 Z M 200 18 C 154 18 116 52 116 98 C 116 191 213 231 311 236 L 311 141 C 311 74 267 18 200 18 Z ",[75,3041],{"id":3042,"className":3043,"d":3044},"gl3Hi1Uc",[79],"M 265 705 C 176 705 110 633 110 544 L 110 434 L 33 434 L 33 396 L 110 396 L 110 79 C 110 60 107 48 100 44 C 93 40 71 39 34 39 L 34 0 L 149 3 C 191 4 234 3 277 0 L 277 39 L 253 39 C 220 39 199 41 192 46 C 185 51 182 63 182 81 L 182 396 L 294 396 L 294 434 L 179 434 L 179 545 C 179 605 210 675 265 675 C 273 675 280 674 287 672 C 272 664 264 650 264 631 C 264 600 279 584 310 584 C 341 584 357 600 357 632 C 357 677 310 705 265 705 Z ",[75,3046],{"id":1355,"className":3047,"d":1357},[79],[75,3049],{"id":1365,"className":3050,"d":1367},[79],[75,3052],{"id":3053,"className":3054,"d":3055},"gl3Hi1VQ",[79],"M 360 130 C 360 206 308 254 205 274 C 158 283 128 294 112 305 C 96 316 88 332 88 351 C 88 398 123 421 193 421 C 261 421 298 383 303 306 C 304 298 309 294 319 294 C 330 294 336 303 336 321 L 336 420 C 336 439 331 448 321 448 C 310 448 295 430 285 422 C 260 439 229 448 193 448 C 104 448 33 408 33 323 C 33 289 47 260 76 237 C 111 208 144 202 207 191 C 272 178 305 149 305 104 C 305 47 270 18 199 18 C 130 18 86 64 67 155 C 64 168 59 175 50 175 C 39 175 33 165 33 146 L 33 17 C 33 -2 38 -11 48 -11 C 51 -11 56 -8 64 -1 C 82 15 74 12 92 30 C 121 3 157 -11 199 -11 C 294 -11 360 38 360 130 Z ",[75,3057],{"id":1375,"className":3058,"d":1377},[79],[75,3060],{"id":3061,"className":3062,"d":3063},"gl3Hi1Vk",[79],"M 418 3 C 441 4 474 3 516 0 L 516 38 C 463 38 443 39 422 67 L 292 235 C 335 288 364 324 380 343 C 408 375 447 392 498 393 L 498 431 C 470 429 442 428 414 428 C 377 428 344 429 315 431 L 315 393 C 334 391 344 382 344 366 C 344 357 339 346 330 334 L 272 262 L 198 357 C 193 364 190 369 190 372 C 190 385 202 392 225 393 L 225 431 L 114 428 C 88 427 56 428 17 431 L 17 393 C 53 393 76 390 86 385 C 96 380 112 363 134 334 L 229 210 C 180 147 150 109 139 96 C 107 57 65 38 12 38 L 12 0 C 49 3 77 4 98 3 L 195 0 L 195 38 C 176 41 167 50 167 65 C 167 69 168 74 170 79 C 176 92 203 126 251 183 L 323 89 C 334 75 340 65 343 59 C 343 46 331 39 307 38 L 307 0 Z ",[75,3065],{"id":3066,"className":3067,"d":3068},"gl3Hi1e4K",[79],"M 137 3 C 155 3 217 0 236 0 C 251 0 258 8 258 23 C 258 32 252 38 239 39 C 210 40 196 50 196 69 C 196 78 205 98 224 128 C 251 173 270 206 283 228 L 526 228 C 526 221 527 207 530 185 C 537 112 541 72 541 67 C 541 48 519 39 474 39 C 455 39 446 31 446 15 C 446 5 452 0 464 0 C 488 0 564 3 588 3 C 608 3 679 0 699 0 C 714 0 721 8 721 24 C 721 34 712 39 694 39 C 666 39 649 41 644 44 C 639 47 635 56 634 71 L 574 689 C 571 708 572 716 551 716 C 539 716 529 710 522 697 L 178 120 C 148 70 108 43 59 39 C 43 38 35 30 35 15 C 35 5 41 0 52 0 C 68 0 121 3 137 3 Z M 492 577 L 522 267 L 307 267 Z ",[75,3070],{"id":2473,"className":3071,"d":2475},[79],[75,3073],{"id":3074,"className":3075,"d":3076},"gl3Hi1foK",[79],"M 594 16 C 594 5 600 0 613 0 L 736 3 C 759 4 840 0 859 0 C 874 0 882 8 882 24 C 882 34 872 39 851 39 C 810 39 790 43 790 52 C 790 55 792 62 795 75 L 927 602 C 932 623 942 636 955 641 C 961 643 979 644 1008 644 C 1033 644 1044 645 1044 668 C 1044 678 1034 683 1013 683 L 882 683 C 853 683 853 681 840 662 L 484 108 L 408 657 C 404 682 402 683 373 683 L 237 683 C 215 683 204 682 204 660 C 204 649 215 644 236 644 C 256 644 297 647 297 631 C 297 629 296 623 293 613 L 167 110 C 158 72 135 49 100 42 C 75 39 42 43 42 16 C 42 5 48 0 60 0 C 78 0 141 3 159 3 C 178 3 242 0 261 0 C 276 0 283 8 283 24 C 283 33 276 38 261 39 C 219 39 198 52 198 78 C 198 82 199 89 202 100 L 332 621 L 415 26 C 418 9 424 0 434 0 C 442 0 450 7 459 20 L 848 627 L 712 82 C 706 60 696 47 681 42 C 675 40 656 39 625 39 C 604 39 594 37 594 16 Z ",[75,3078],{"id":3079,"className":3080,"d":3081},"gl3Hi1fwK",[79],"M 702 263 C 727 320 740 377 740 435 C 740 513 717 577 670 628 C 623 679 561 705 484 705 C 431 705 378 692 325 666 C 217 613 138 529 87 415 C 62 359 49 303 49 246 C 49 167 72 103 119 53 C 166 3 228 -22 306 -22 C 358 -22 410 -9 463 16 C 572 69 651 152 702 263 Z M 644 474 C 644 341 584 208 520 129 C 476 74 400 12 312 12 C 205 12 149 105 149 218 C 149 253 156 300 169 357 C 196 475 264 579 353 633 C 396 659 438 672 478 672 C 586 672 644 586 644 474 Z ",[75,3083],{"id":2478,"className":3084,"d":2480},[79],[75,3086],{"id":2483,"className":3087,"d":2485},[79],[75,3089],{"id":3090,"className":3091,"d":3092},"gl3Hi1QsL",[79],"M 429 632 L 370 389 C 349 426 320 445 281 445 C 216 445 159 412 109 345 C 63 283 40 218 40 151 C 40 63 91 -11 175 -11 C 218 -11 260 12 300 59 C 311 21 345 -11 392 -11 C 461 -11 483 71 498 145 C 498 154 493 159 483 159 C 474 159 468 152 465 138 C 445 59 421 19 394 19 C 377 19 368 33 368 60 C 368 75 370 91 374 107 L 516 679 L 516 683 C 513 690 507 694 499 694 C 476 694 434 690 374 683 C 363 682 357 674 357 660 C 357 650 366 645 384 645 C 403 645 429 646 429 632 Z M 341 376 C 351 355 356 341 356 332 C 355 328 354 323 353 316 L 304 122 C 301 111 294 99 285 87 C 248 42 212 19 177 19 C 138 19 118 49 118 108 C 118 132 124 168 136 216 C 157 301 187 359 224 390 C 244 407 263 415 282 415 C 309 415 329 402 341 376 Z ",[75,3094],{"id":103,"className":3095,"d":105},[79],[75,3097],{"id":3098,"className":3099,"d":3100},"gl3Hi1dMT",[79],"M 223 629 C 223 625 222 618 219 608 C 171 417 125 224 76 34 C 75 30 75 27 75 26 C 75 2 87 -10 112 -10 C 142 -10 154 9 163 38 L 196 169 C 207 217 218 271 241 305 C 286 374 337 408 395 408 C 431 408 449 386 449 343 C 449 304 430 235 392 134 C 382 107 377 89 377 79 C 377 25 420 -10 474 -10 C 521 -10 559 14 586 63 C 609 104 620 131 620 145 C 620 156 613 162 600 162 C 593 161 586 156 579 146 C 578 145 578 143 578 142 C 555 65 522 26 477 26 C 463 26 456 36 456 56 C 456 71 464 97 479 135 C 514 224 531 288 531 327 C 531 405 487 444 399 444 C 342 444 291 420 246 372 L 317 659 C 318 661 319 667 321 676 C 321 688 315 694 302 694 L 175 684 C 158 683 150 674 150 657 C 150 646 160 640 179 640 C 197 640 223 641 223 629 Z ",[75,3102],{"id":3103,"className":3104,"d":3105},"gl3Hi1fUW",[79],"M 798 -1245 C 815 -1245 823 -1237 823 -1220 C 823 -1212 820 -1205 813 -1200 C 732 -1139 659 -1043 592 -910 C 473 -673 385 -340 385 1 L 385 499 C 385 654 403 810 438 967 C 500 1247 630 1562 813 1700 C 820 1705 823 1712 823 1720 C 823 1737 815 1745 798 1745 C 793 1745 788 1743 784 1740 C 697 1673 614 1574 536 1441 C 398 1205 277 850 277 499 L 277 1 C 277 -329 388 -667 512 -899 C 594 -1050 685 -1164 784 -1240 C 788 -1243 793 -1245 798 -1245 Z ",[75,3107],{"id":3108,"className":3109,"d":3110},"gl3Hi1fYW",[79],"M 598 1 L 598 499 C 598 829 487 1167 363 1399 C 281 1550 190 1664 91 1740 C 87 1743 82 1745 77 1745 C 60 1745 52 1737 52 1720 C 52 1712 55 1705 62 1700 C 143 1639 216 1543 283 1410 C 402 1173 490 840 490 499 L 490 1 C 490 -154 472 -310 437 -467 C 375 -747 245 -1062 62 -1200 C 55 -1205 52 -1212 52 -1220 C 52 -1237 60 -1245 77 -1245 C 82 -1245 87 -1243 91 -1240 C 178 -1173 261 -1074 339 -941 C 477 -705 598 -350 598 1 Z ",[75,3112],{"id":3113,"className":3114,"d":3115},"gl3Hi1bwX",[79],"M 1013 807 C 1018 817 1020 823 1020 826 C 1020 842 1012 850 996 850 C 986 850 978 843 971 828 L 463 -233 L 253 256 L 108 144 L 128 122 L 198 175 L 423 -350 C 454 -350 461 -349 470 -328 Z ",[70,3117],{"className":3118},[134],[65,3120],{"type":67},[138,3122,3127],{"className":3123,"transform":142,"dataTid":3124,"dataPageWidth":3125,"dataPageHeight":3126},[141],"p7Xr3kjiK6jQIHBIjvXdyII","185","33",[138,3128,3131],{"dataTid":3129,"className":3130},"g7Xr3kjiK6jQIHBIjvXdyII",[150],[138,3132,3134],{"transform":153,"dataTid":3133},"pNbrfDMpRALQ0aGr9dum5aQ",[138,3135,3138,3500],{"dataTid":3136,"className":3137},"gNbrfDMpRALQ0aGr9dum5aQ",[150],[138,3139,3141],{"transform":153,"dataTid":3140},"pD7Z+nZeBJMs3J74wd6EDrY",[138,3142,3145,3165,3180,3221,3240,3377,3392,3412,3430,3450,3465,3480,3495],{"dataTid":3143,"className":3144},"gD7Z+nZeBJMs3J74wd6EDrY",[150],[138,3146,3149],{"transform":3147,"dataTid":3148},"translate(0.000,19.195)","p\u002FKVdl69XseEbGxfsocN\u002Fzk",[138,3150,3153,3156],{"dataTid":3151,"className":3152,"fill":175,"transform":176},"g\u002FKVdl69XseEbGxfsocN\u002Fzk",[174],[178,3154],{"x":180,"y":180,"href":3155},"#gl3Hi1e4K",[138,3157,3158],{"transform":184},[186,3159,3161],{"x":180,"y":188,"width":3160,"height":190},"46.88",[192,3162,3164],{"className":3163,"style":196},[195],"𝐴",[138,3166,3168],{"transform":3167,"dataTid":1442},"translate(11.306,19.195)",[138,3169,3171,3173],{"dataTid":1445,"className":3170,"fill":175,"transform":176},[174],[178,3172],{"x":180,"y":180,"href":1449},[138,3174,3175],{"transform":184},[186,3176,3177],{"x":180,"y":188,"width":1454,"height":190},[192,3178,946],{"className":3179,"style":196},[195],[138,3181,3184],{"transform":3182,"dataTid":3183},"translate(22.919,19.195)","pO+eGEwODvRU7lWYl+jAkXo",[138,3185,3188,3191,3194,3198,3201,3204,3208,3212],{"dataTid":3186,"className":3187,"fill":175,"transform":176},"gO+eGEwODvRU7lWYl+jAkXo",[174],[178,3189],{"x":180,"y":180,"href":3190},"#gl3Hi1VQ",[178,3192],{"x":3193,"y":180,"href":1484},"394",[178,3195],{"x":3196,"y":180,"href":3197},"894","#gl3Hi1Uc",[178,3199],{"x":3200,"y":180,"href":1636},"1200",[178,3202],{"x":3203,"y":180,"href":1492},"1589",[178,3205],{"x":3206,"y":180,"href":3207},"2422","#gl3Hi1UI",[178,3209],{"x":3210,"y":180,"href":3211},"2922","#gl3Hi1Vk",[138,3213,3214],{"transform":184},[186,3215,3217],{"x":180,"y":188,"width":3216,"height":190},"215.62",[192,3218,3220],{"className":3219,"style":196},[195],"softmax",[138,3222,3225],{"transform":3223,"dataTid":3224},"translate(60.869,19.195)","poEsihrMsCDQzkCW+ZETc8E",[138,3226,3229,3232],{"dataTid":3227,"className":3228,"fill":175,"transform":176},"goEsihrMsCDQzkCW+ZETc8E",[174],[178,3230],{"x":180,"y":180,"href":3231},"#gl3Hi1fUW",[138,3233,3234],{"transform":184},[186,3235,3237],{"x":180,"y":188,"width":3236,"height":190},"54.69",[192,3238,238],{"className":3239,"style":196},[195],[138,3241,3244],{"transform":3242,"dataTid":3243},"translate(70.494,2.550)","pqaMyKKCm6nAJ40pSz7qAwg",[138,3245,3248,3263,3278,3293,3312,3327,3346,3366],{"dataTid":3246,"className":3247},"gqaMyKKCm6nAJ40pSz7qAwg",[150],[138,3249,3251],{"transform":3250,"dataTid":2538},"translate(1.100,9.198)",[138,3252,3254,3256],{"dataTid":2541,"className":3253,"fill":175,"transform":176},[174],[178,3255],{"x":180,"y":180,"href":2545},[138,3257,3258],{"transform":184},[186,3259,3260],{"x":180,"y":188,"width":2550,"height":190},[192,3261,2554],{"className":3262,"style":196},[195],[138,3264,3266],{"transform":3265,"dataTid":2627},"translate(9.801,9.198)",[138,3267,3269,3271],{"dataTid":2630,"className":3268,"fill":175,"transform":176},[174],[178,3270],{"x":180,"y":180,"href":2634},[138,3272,3273],{"transform":184},[186,3274,3275],{"x":180,"y":188,"width":2639,"height":190},[192,3276,2643],{"className":3277,"style":196},[195],[138,3279,3281],{"transform":3280,"dataTid":262},"translate(19.888,5.205)",[138,3282,3284,3286],{"dataTid":265,"className":3283,"fill":175,"transform":206},[174],[178,3285],{"x":180,"y":180,"href":269},[138,3287,3288],{"transform":184},[186,3289,3290],{"x":180,"y":188,"width":274,"height":190},[192,3291,278],{"className":3292,"style":196},[195],[138,3294,3297],{"transform":3295,"dataTid":3296},"translate(2.708,25.357)","poUm8rZfdd1cLH+tiJdp5YA",[138,3298,3301,3304],{"dataTid":3299,"className":3300,"fill":175,"transform":176},"goUm8rZfdd1cLH+tiJdp5YA",[174],[178,3302],{"x":180,"y":180,"href":3303},"#gl3Hi1bwX",[138,3305,3306],{"transform":184},[186,3307,3308],{"x":180,"y":188,"width":190,"height":190},[192,3309,3311],{"className":3310,"style":196},[195],"√",[138,3313,3316],{"transform":3314,"dataTid":3315},"translate(13.708,16.271)","pdP2Kbwg\u002FVQoqZHrF8tjnI4",[138,3317,3319],{"dataTid":3318},"gdP2Kbwg\u002FVQoqZHrF8tjnI4",[75,3320],{"className":3321,"d":3323,"strokeWidth":3324,"strokeMiterLimit":970,"fillRule":3325,"fill":3326,"stroke":175},[3322],"typst-shape","M 0 0 L 11.4873 0 ","0.528","nonzero","none",[138,3328,3331],{"transform":3329,"dataTid":3330},"translate(13.708,25.566)","pcs7iuBS0TA4jxSbb6T+Sz4",[138,3332,3335,3338],{"dataTid":3333,"className":3334,"fill":175,"transform":176},"gcs7iuBS0TA4jxSbb6T+Sz4",[174],[178,3336],{"x":180,"y":180,"href":3337},"#gl3Hi1QsL",[138,3339,3340],{"transform":184},[186,3341,3342],{"x":180,"y":188,"width":214,"height":190},[192,3343,3345],{"className":3344,"style":196},[195],"𝑑",[138,3347,3350],{"transform":3348,"dataTid":3349},"translate(19.428,28.283)","pfj\u002FOSeBAGe8uj6vOHu\u002FGlE",[138,3351,3354,3357],{"dataTid":3352,"className":3353,"fill":175,"transform":206},"gfj\u002FOSeBAGe8uj6vOHu\u002FGlE",[174],[178,3355],{"x":180,"y":180,"href":3356},"#gl3Hi1dMT",[138,3358,3359],{"transform":184},[186,3360,3362],{"x":180,"y":188,"width":3361,"height":190},"41.81",[192,3363,3365],{"className":3364,"style":196},[195],"ℎ",[138,3367,3370],{"transform":3368,"dataTid":3369},"translate(1.100,13.895)","pXyA7Vw+YGaAnz3st05oDVg",[138,3371,3373],{"dataTid":3372},"gXyA7Vw+YGaAnz3st05oDVg",[75,3374],{"className":3375,"d":3376,"strokeWidth":3324,"strokeMiterLimit":970,"fillRule":3325,"fill":3326,"stroke":175},[3322],"M 0 0 L 25.7026 0 ",[138,3378,3380],{"transform":3379,"dataTid":1607},"translate(100.841,19.195)",[138,3381,3383,3385],{"dataTid":1610,"className":3382,"fill":175,"transform":176},[174],[178,3384],{"x":180,"y":180,"href":1614},[138,3386,3387],{"transform":184},[186,3388,3389],{"x":180,"y":188,"width":1454,"height":190},[192,3390,298],{"className":3391,"style":196},[195],[138,3393,3396],{"transform":3394,"dataTid":3395},"translate(111.844,19.195)","pS6AjlB4F4L0yX8AEZwgfSw",[138,3397,3400,3403],{"dataTid":3398,"className":3399,"fill":175,"transform":176},"gS6AjlB4F4L0yX8AEZwgfSw",[174],[178,3401],{"x":180,"y":180,"href":3402},"#gl3Hi1foK",[138,3404,3405],{"transform":184},[186,3406,3408],{"x":180,"y":188,"width":3407,"height":190},"67.00",[192,3409,3411],{"className":3410,"style":196},[195],"𝑀",[138,3413,3416],{"transform":3414,"dataTid":3415},"translate(123.636,19.195)","pbymWxyxE24cANafQCBqh4o",[138,3417,3420,3423],{"dataTid":3418,"className":3419,"fill":175,"transform":176},"gbymWxyxE24cANafQCBqh4o",[174],[178,3421],{"x":180,"y":180,"href":3422},"#gl3Hi1fYW",[138,3424,3425],{"transform":184},[186,3426,3427],{"x":180,"y":188,"width":3236,"height":190},[192,3428,472],{"className":3429,"style":196},[195],[138,3431,3434],{"transform":3432,"dataTid":3433},"translate(144.261,19.195)","pt6WrT99hftUtcpA9SgQVfc",[138,3435,3438,3441],{"dataTid":3436,"className":3437,"fill":175,"transform":176},"gt6WrT99hftUtcpA9SgQVfc",[174],[178,3439],{"x":180,"y":180,"href":3440},"#gl3Hi1fwK",[138,3442,3443],{"transform":184},[186,3444,3446],{"x":180,"y":188,"width":3445,"height":190},"48.00",[192,3447,3449],{"className":3448,"style":196},[195],"𝑂",[138,3451,3453],{"transform":3452,"dataTid":1442},"translate(155.764,19.195)",[138,3454,3456,3458],{"dataTid":1445,"className":3455,"fill":175,"transform":176},[174],[178,3457],{"x":180,"y":180,"href":1449},[138,3459,3460],{"transform":184},[186,3461,3462],{"x":180,"y":188,"width":1454,"height":190},[192,3463,946],{"className":3464,"style":196},[195],[138,3466,3468],{"transform":3467,"dataTid":3148},"translate(167.378,19.195)",[138,3469,3471,3473],{"dataTid":3151,"className":3470,"fill":175,"transform":176},[174],[178,3472],{"x":180,"y":180,"href":3155},[138,3474,3475],{"transform":184},[186,3476,3477],{"x":180,"y":188,"width":3160,"height":190},[192,3478,3164],{"className":3479,"style":196},[195],[138,3481,3483],{"transform":3482,"dataTid":2711},"translate(175.628,19.195)",[138,3484,3486,3488],{"dataTid":2714,"className":3485,"fill":175,"transform":176},[174],[178,3487],{"x":180,"y":180,"href":2718},[138,3489,3490],{"transform":184},[186,3491,3492],{"x":180,"y":188,"width":2723,"height":190},[192,3493,2727],{"className":3494,"style":196},[195],[138,3496,3497],{"transform":153,"dataTid":475},[138,3498],{"dataTid":478,"className":3499,"dataHint":15},[480],[138,3501,3502],{"transform":153,"dataTid":475},[138,3503],{"dataTid":478,"className":3504,"dataHint":15},[480],[11,3506,3507],{},"We will follow the recorded scores from layer 15 and query head 1, with both indices starting at zero. These values come from the Transformers reference run.",[11,3509,3510],{},"The first matrix multiplication takes the 47 queries and 47 keys of the head and computes one dot product per pair. That gives a grid of 47 by 47 scores. Each score is divided by the square root of the head size, √128, about 11.3, so the scores don't grow with the size of the head. In Foundry this is one chain of operations. The queries are reshaped so that the 3 heads sharing a key head sit together, and then they are multiplied by the transposed keys in one go.",[898,3512,3514],{"className":900,"code":3513,"language":902,"meta":903,"style":903},"let scores = query\n    .contiguous(device)?\n    .reshape(Shape::try_from([batch, kv_heads, rows, head_dim])?)?\n    .matmul(device, &key_t)?\n    .reshape(Shape::try_from([batch.saturating_mul(kv_heads), group, seq_len, kv_length])?)?\n    .to_dtype(device, DType::F32)?\n    .mul(device, scale)?\n    .add(device, attn_mask)?;\n",[868,3515,3516,3528,3539,3562,3578,3609,3627,3638],{"__ignoreMap":903},[907,3517,3518,3520,3523,3525],{"class":909,"line":910},[907,3519,940],{"class":913},[907,3521,3522],{"class":926}," scores ",[907,3524,946],{"class":913},[907,3526,3527],{"class":926}," query\n",[907,3529,3530,3532,3535,3537],{"class":909,"line":930},[907,3531,2308],{"class":913},[907,3533,3534],{"class":916},"contiguous",[907,3536,2382],{"class":926},[907,3538,2321],{"class":913},[907,3540,3541,3543,3545,3547,3549,3551,3553,3556,3558,3560],{"class":909,"line":937},[907,3542,2308],{"class":913},[907,3544,1128],{"class":916},[907,3546,238],{"class":926},[907,3548,1133],{"class":916},[907,3550,920],{"class":913},[907,3552,1138],{"class":916},[907,3554,3555],{"class":926},"([batch, kv_heads, rows, head_dim])",[907,3557,965],{"class":913},[907,3559,472],{"class":926},[907,3561,2321],{"class":913},[907,3563,3564,3566,3569,3571,3573,3576],{"class":909,"line":970},[907,3565,2308],{"class":913},[907,3567,3568],{"class":916},"matmul",[907,3570,1190],{"class":926},[907,3572,1198],{"class":913},[907,3574,3575],{"class":926},"key_t)",[907,3577,2321],{"class":913},[907,3579,3580,3582,3584,3586,3588,3590,3592,3595,3597,3600,3603,3605,3607],{"class":909,"line":985},[907,3581,2308],{"class":913},[907,3583,1128],{"class":916},[907,3585,238],{"class":926},[907,3587,1133],{"class":916},[907,3589,920],{"class":913},[907,3591,1138],{"class":916},[907,3593,3594],{"class":926},"([batch",[907,3596,1072],{"class":913},[907,3598,3599],{"class":916},"saturating_mul",[907,3601,3602],{"class":926},"(kv_heads), group, seq_len, kv_length])",[907,3604,965],{"class":913},[907,3606,472],{"class":926},[907,3608,2321],{"class":913},[907,3610,3611,3613,3615,3617,3619,3621,3623,3625],{"class":909,"line":1001},[907,3612,2308],{"class":913},[907,3614,2275],{"class":916},[907,3616,1190],{"class":926},[907,3618,2280],{"class":916},[907,3620,920],{"class":913},[907,3622,2285],{"class":916},[907,3624,472],{"class":926},[907,3626,2321],{"class":913},[907,3628,3629,3631,3633,3636],{"class":909,"line":1014},[907,3630,2308],{"class":913},[907,3632,2311],{"class":916},[907,3634,3635],{"class":926},"(device, scale)",[907,3637,2321],{"class":913},[907,3639,3640,3642,3644,3647,3649],{"class":909,"line":1031},[907,3641,2308],{"class":913},[907,3643,2162],{"class":916},[907,3645,3646],{"class":926},"(device, attn_mask)",[907,3648,965],{"class":913},[907,3650,927],{"class":926},[11,3652,3653],{},"The last line adds the causal mask M. A position may only look at itself and at earlier positions. Every score that points to a later position gets minus infinity. Foundry builds the mask as a grid of zeros and minus infinities.",[898,3655,3657],{"className":900,"code":3656,"language":902,"meta":903,"style":903},"(0..kv_length).map(move |key_position| {\n    if key_position \u003C= query_position { 0.0_f32 } else { f32::NEG_INFINITY }\n})\n",[868,3658,3659,3692,3731],{"__ignoreMap":903},[907,3660,3661,3663,3665,3668,3671,3673,3676,3678,3681,3684,3687,3689],{"class":909,"line":910},[907,3662,238],{"class":926},[907,3664,180],{"class":1007},[907,3666,3667],{"class":913},"..",[907,3669,3670],{"class":926},"kv_length)",[907,3672,1072],{"class":913},[907,3674,3675],{"class":916},"map",[907,3677,238],{"class":926},[907,3679,3680],{"class":913},"move",[907,3682,3683],{"class":913}," |",[907,3685,3686],{"class":926},"key_position",[907,3688,338],{"class":913},[907,3690,3691],{"class":926}," {\n",[907,3693,3694,3697,3700,3703,3706,3709,3712,3715,3718,3721,3723,3725,3728],{"class":909,"line":930},[907,3695,3696],{"class":913},"    if",[907,3698,3699],{"class":926}," key_position ",[907,3701,3702],{"class":913},"\u003C=",[907,3704,3705],{"class":926}," query_position { ",[907,3707,3708],{"class":1007},"0.0_",[907,3710,3711],{"class":916},"f32",[907,3713,3714],{"class":926}," } ",[907,3716,3717],{"class":913},"else",[907,3719,3720],{"class":926}," { ",[907,3722,3711],{"class":916},[907,3724,920],{"class":913},[907,3726,3727],{"class":1007},"NEG_INFINITY",[907,3729,3730],{"class":926}," }\n",[907,3732,3733],{"class":909,"line":937},[907,3734,3735],{"class":926},"})\n",[11,3737,3738,3739,3803],{},"Softmax turns each row of scores into weights between 0 and 1 that add up to 1. It raises ",[51,3740,3742,3744,3752,3755,3757],{"style":491,"className":3741,"viewBox":493,"width":494,"height":495,"dataWidth":496,"dataHeight":497,"xmlns":61,"xmlnsXLink":62,"xmlns:h5":63},[55],[65,3743,68],{"type":67},[70,3745,3747],{"className":3746},[73],[75,3748],{"id":3749,"className":3750,"d":3751},"gl3Hi1QwL",[79],"M 124 129 C 124 153 129 186 139 227 L 188 227 C 253 227 303 235 339 250 C 372 264 394 284 405 309 C 412 326 415 342 415 355 C 415 410 363 442 307 442 C 268 442 229 432 190 412 C 113 372 46 281 46 171 C 46 69 105 -11 204 -11 C 257 -11 304 2 345 27 C 379 48 404 69 420 90 C 427 99 430 106 430 109 C 430 120 425 126 414 126 C 409 126 404 122 398 114 C 365 70 324 42 277 30 C 246 22 223 18 206 18 C 149 18 124 72 124 129 Z M 375 355 C 375 289 311 256 182 256 L 147 256 C 166 322 194 366 232 387 C 262 404 287 413 307 413 C 343 413 375 391 375 355 Z ",[70,3753],{"className":3754},[134],[65,3756],{"type":67},[138,3758,3761],{"className":3759,"transform":142,"dataTid":3760,"dataPageWidth":517,"dataPageHeight":518},[141],"pQ\u002FhXJIg\u002FlngojZtIihEEoQ",[138,3762,3765],{"dataTid":3763,"className":3764},"gQ\u002FhXJIg\u002FlngojZtIihEEoQ",[150],[138,3766,3768],{"transform":153,"dataTid":3767},"p6\u002FvocT9Njp08E6NpeKV26Q",[138,3769,3772,3798],{"dataTid":3770,"className":3771},"g6\u002FvocT9Njp08E6NpeKV26Q",[150],[138,3773,3775],{"transform":153,"dataTid":3774},"pUSobjb2LKNY6St50p2Z8Y4",[138,3776,3779],{"dataTid":3777,"className":3778},"gUSobjb2LKNY6St50p2Z8Y4",[150],[138,3780,3782],{"transform":539,"dataTid":3781},"paUMxrJ8Gn9UhY9MNG1j9bc",[138,3783,3786,3789],{"dataTid":3784,"className":3785,"fill":175,"transform":176},"gaUMxrJ8Gn9UhY9MNG1j9bc",[174],[178,3787],{"x":180,"y":180,"href":3788},"#gl3Hi1QwL",[138,3790,3791],{"transform":184},[186,3792,3794],{"x":180,"y":188,"width":3793,"height":190},"29.12",[192,3795,3797],{"className":3796,"style":196},[195],"𝑒",[138,3799,3800],{"transform":153,"dataTid":475},[138,3801],{"dataTid":478,"className":3802,"dataHint":15},[480]," to the power of each score and divides by the sum. A higher score gets a larger share of the weight.",[51,3805,3812,3814,3900,3903,3905],{"style":53,"className":3806,"viewBox":3807,"width":3808,"height":3809,"dataWidth":3810,"dataHeight":3811,"xmlns":61,"xmlnsXLink":62,"xmlns:h5":63},[55],"0 0 187.000 29.000","17em","2.6363636363636362em","187.000","29.000",[65,3813,68],{"type":67},[70,3815,3817,3820,3823,3826,3829,3832,3835,3838,3841,3844,3847,3852,3855,3860,3865,3870,3875,3880,3885,3890,3895],{"className":3816},[73],[75,3818],{"id":1300,"className":3819,"d":1302},[79],[75,3821],{"id":1305,"className":3822,"d":1307},[79],[75,3824],{"id":1315,"className":3825,"d":1317},[79],[75,3827],{"id":3037,"className":3828,"d":3039},[79],[75,3830],{"id":3042,"className":3831,"d":3044},[79],[75,3833],{"id":1355,"className":3834,"d":1357},[79],[75,3836],{"id":1365,"className":3837,"d":1367},[79],[75,3839],{"id":3053,"className":3840,"d":3055},[79],[75,3842],{"id":1375,"className":3843,"d":1377},[79],[75,3845],{"id":3061,"className":3846,"d":3063},[79],[75,3848],{"id":3849,"className":3850,"d":3851},"gl3Hi1dQD",[79],"M 56 713 L 417 216 L 67 -217 C 60 -225 57 -231 57 -235 C 57 -245 67 -250 88 -250 L 913 -250 L 1001 5 L 967 5 C 954 -33 933 -66 902 -95 C 825 -166 719 -188 565 -188 L 152 -188 L 490 232 C 497 240 500 246 500 250 C 500 254 498 259 493 266 L 175 702 L 560 702 C 682 702 764 691 854 644 C 907 616 944 571 967 510 L 1001 510 L 913 750 L 88 750 C 57 750 56 746 56 713 Z ",[75,3853],{"id":3749,"className":3854,"d":3751},[79],[75,3856],{"id":3857,"className":3858,"d":3859},"gl3Hi1RML",[79],"M 657 442 C 596 442 543 412 498 353 C 489 412 451 442 383 442 C 324 442 273 416 231 363 C 223 407 188 442 137 442 C 96 442 66 409 45 344 C 34 312 29 293 29 287 C 29 278 34 273 45 273 C 50 273 53 274 56 276 C 61 285 64 292 66 299 C 84 375 107 413 134 413 C 152 413 161 399 161 371 C 161 358 156 331 145 290 L 88 63 C 84 51 79 25 79 19 C 79 -1 90 -11 111 -11 C 131 -11 145 -1 152 19 C 153 24 160 49 171 92 L 192 181 L 222 295 C 233 318 250 341 272 365 C 301 397 337 413 380 413 C 413 413 429 391 429 348 C 429 335 424 308 414 267 L 387 153 C 380 124 364 64 356 32 C 355 25 354 21 354 19 C 354 -1 365 -11 387 -11 C 398 -11 406 -8 413 -1 C 428 14 429 21 435 48 L 494 285 C 497 298 511 321 535 353 C 565 393 605 413 654 413 C 687 413 703 391 703 348 C 703 309 683 236 642 129 C 633 106 629 87 629 74 C 629 25 666 -11 714 -11 C 759 -11 793 14 818 64 C 838 104 848 131 848 144 C 848 153 843 158 832 158 C 825 157 818 148 813 137 C 791 58 759 18 716 18 C 703 18 696 28 696 47 C 696 62 702 84 714 115 C 755 222 775 295 775 333 C 775 404 728 442 657 442 Z ",[75,3861],{"id":3862,"className":3863,"d":3864},"gl3Hi1SAL",[79],"M 162 92 L 145 90 C 193 137 230 171 255 194 L 351 283 C 374 304 398 330 421 359 C 452 397 467 420 467 428 C 467 437 462 442 452 442 C 445 442 439 438 434 429 C 409 388 386 368 364 368 C 342 368 326 386 317 398 C 293 427 269 442 246 442 C 220 442 193 430 165 404 C 137 378 123 352 123 326 C 123 316 128 311 139 311 C 146 311 152 316 156 325 C 167 354 194 368 235 368 C 249 368 269 363 296 354 C 320 345 343 340 364 339 C 308 284 265 244 234 217 L 145 134 C 130 119 111 99 90 74 C 59 35 43 12 43 3 C 43 -6 48 -11 59 -11 C 65 -11 71 -6 78 4 C 103 43 130 63 158 63 C 174 63 192 51 212 26 C 232 1 255 -11 278 -11 C 312 -11 346 7 382 42 C 418 77 436 111 436 144 C 436 153 431 158 420 158 C 413 158 407 153 402 143 C 387 100 344 63 288 63 C 264 63 186 92 162 92 Z ",[75,3866],{"id":3867,"className":3868,"d":3869},"gl3Hi1Z4T",[79],"M 397 621 C 397 648 376 667 350 667 C 318 667 287 636 287 604 C 287 578 306 557 333 557 C 365 557 397 589 397 621 Z M 252 445 C 203 445 163 422 132 376 C 105 337 91 307 91 287 C 91 276 99 270 115 270 C 128 270 137 277 142 292 C 156 340 195 403 248 403 C 265 403 274 392 274 370 C 274 359 272 347 268 335 C 240 265 211 195 182 126 C 175 105 171 89 171 80 C 171 24 224 -11 282 -11 C 331 -11 371 12 402 58 C 429 98 443 127 443 146 C 443 158 435 164 419 164 C 407 164 399 160 396 152 C 393 144 392 140 391 138 C 376 92 337 31 286 31 C 269 31 260 42 260 64 C 260 75 264 92 273 113 L 349 303 C 358 326 363 343 363 354 C 363 411 309 445 252 445 Z ",[75,3871],{"id":3872,"className":3873,"d":3874},"gl3Hi1Z8T",[79],"M 510 620 C 510 647 490 667 463 667 C 431 667 398 636 398 604 C 398 578 419 557 445 557 C 478 557 510 587 510 620 Z M 350 444 C 293 444 245 420 206 373 C 171 332 154 303 154 286 C 154 275 162 269 179 269 C 190 269 199 274 204 285 C 241 363 289 402 346 402 C 371 402 383 386 383 354 C 383 345 382 336 380 329 L 288 -42 C 272 -101 221 -163 151 -163 C 143 -163 135 -162 126 -161 C 142 -148 150 -131 150 -110 C 150 -83 128 -65 102 -65 C 67 -65 37 -98 37 -133 C 37 -183 94 -205 152 -205 C 179 -205 206 -201 235 -192 C 303 -172 364 -113 381 -44 L 470 310 C 473 323 475 335 475 344 C 475 405 415 444 350 444 Z ",[75,3876],{"id":3877,"className":3878,"d":3879},"gl3Hi1dQT",[79],"M 325 621 C 325 645 306 663 282 663 C 252 663 223 635 223 605 C 223 581 242 563 266 563 C 296 563 325 591 325 621 Z M 192 444 C 145 444 108 420 81 372 C 58 332 47 304 47 288 C 47 277 54 272 67 272 C 78 272 85 278 89 290 C 112 369 145 408 189 408 C 203 408 210 398 210 377 C 210 364 200 332 180 280 L 118 121 C 112 105 109 91 109 78 C 109 26 154 -10 206 -10 C 253 -10 289 14 316 63 C 339 103 350 130 350 145 C 350 156 343 162 330 162 C 324 161 317 156 310 146 C 309 145 308 143 308 142 C 285 65 252 26 209 26 C 195 26 188 36 188 56 C 188 71 193 92 204 118 L 283 323 C 287 334 289 345 289 356 C 289 408 244 444 192 444 Z ",[75,3881],{"id":3882,"className":3883,"d":3884},"gl3Hi1dUT",[79],"M 439 621 C 439 645 420 663 396 663 C 367 663 337 634 337 605 C 337 581 356 563 380 563 C 410 563 439 591 439 621 Z M 294 443 C 238 443 189 406 163 371 C 133 328 118 300 118 287 C 118 276 125 271 138 271 C 148 271 156 278 162 291 C 173 315 186 336 201 355 C 230 390 260 407 291 407 C 310 407 320 392 320 363 C 320 351 319 341 317 334 L 223 -45 C 209 -102 164 -168 100 -168 C 91 -168 83 -167 75 -165 C 93 -154 102 -137 102 -116 C 102 -89 88 -75 60 -75 C 26 -75 0 -104 0 -138 C 0 -184 50 -204 101 -204 C 124 -204 149 -200 175 -191 C 237 -170 293 -111 309 -44 L 398 310 C 401 323 403 335 403 346 C 403 404 352 443 294 443 Z ",[75,3886],{"id":3887,"className":3888,"d":3889},"gl3Hi1dgT",[79],"M 165 441 C 132 441 104 423 83 386 C 69 361 58 335 51 306 C 48 295 47 289 47 287 C 47 276 54 270 68 270 C 87 270 88 281 93 302 C 110 371 133 406 162 406 C 181 406 191 391 191 361 C 191 350 186 324 175 281 L 121 67 C 114 40 111 26 111 25 C 111 2 124 -10 149 -10 C 168 -10 182 -2 191 14 C 194 20 200 41 209 76 L 231 164 C 243 211 250 237 252 243 C 260 269 272 294 289 317 C 331 376 380 406 435 406 C 471 406 489 384 489 341 C 489 324 483 294 472 249 L 431 84 C 422 47 417 28 417 25 C 417 2 429 -10 454 -10 C 473 -10 488 -2 497 14 C 500 20 506 41 515 76 L 537 164 C 553 226 556 263 584 303 C 629 372 680 406 739 406 C 775 406 793 384 793 341 C 793 302 774 233 735 133 C 725 107 720 89 720 78 C 720 24 764 -10 818 -10 C 866 -10 904 14 931 62 C 954 103 965 130 965 144 C 965 155 958 161 945 161 C 938 160 928 152 922 141 C 899 64 866 25 821 25 C 807 25 800 35 800 56 C 800 70 808 96 823 134 C 858 223 876 287 876 325 C 876 402 832 441 743 441 C 678 441 621 410 570 348 C 560 410 516 441 439 441 C 374 441 319 413 274 357 C 265 406 222 441 165 441 Z ",[75,3891],{"id":3892,"className":3893,"d":3894},"gl3Hi1eUT",[79],"M 402 336 C 375 309 334 273 276 227 C 218 181 176 145 150 119 C 128 97 112 80 101 67 C 74 36 61 15 61 6 C 61 -5 68 -10 82 -10 C 90 -10 97 -6 102 2 C 129 41 158 60 189 60 C 204 60 220 52 238 37 C 263 15 278 3 282 0 C 294 -7 308 -10 323 -10 C 356 -10 387 1 416 23 C 447 47 498 101 498 146 C 498 157 491 163 478 163 C 468 163 461 157 456 145 C 441 104 391 75 335 75 C 308 75 221 96 193 95 L 180 95 C 211 125 250 159 299 198 C 365 251 406 285 423 302 C 488 367 521 408 521 425 C 521 436 514 441 500 441 C 492 441 485 437 480 429 C 455 390 431 371 409 371 C 395 371 375 383 348 406 C 321 429 297 441 275 441 C 246 441 217 428 186 402 C 155 376 140 350 140 322 C 140 311 147 306 160 306 C 168 306 175 310 180 319 C 198 348 214 356 263 356 C 290 356 376 337 402 336 Z ",[75,3896],{"id":3897,"className":3898,"d":3899},"gl3Hi1aAb",[79],"M 696 273 L 82 273 C 62 273 56 263 56 250 C 56 237 62 227 82 227 L 696 227 C 716 227 722 237 722 250 C 722 262 716 273 696 273 Z ",[70,3901],{"className":3902},[134],[65,3904],{"type":67},[138,3906,3911],{"className":3907,"transform":142,"dataTid":3908,"dataPageWidth":3909,"dataPageHeight":3910},[141],"p6DTsxjpAJIAa1wJ5\u002FiPA98","187","29",[138,3912,3915],{"dataTid":3913,"className":3914},"g6DTsxjpAJIAa1wJ5\u002FiPA98",[150],[138,3916,3918],{"transform":153,"dataTid":3917},"pw4l5PThfQv8xliBuaNFVao",[138,3919,3922,4389],{"dataTid":3920,"className":3921},"gw4l5PThfQv8xliBuaNFVao",[150],[138,3923,3925],{"transform":153,"dataTid":3924},"pQtmZb6b8Z3cJqdal3VcLLI",[138,3926,3929,3956,3971,3991,4006,4026,4041,4280,4299,4314,4339,4354,4369,4384],{"dataTid":3927,"className":3928},"gQtmZb6b8Z3cJqdal3VcLLI",[150],[138,3930,3932],{"transform":3931,"dataTid":3183},"translate(0.000,14.836)",[138,3933,3935,3937,3939,3941,3943,3945,3947,3949],{"dataTid":3186,"className":3934,"fill":175,"transform":176},[174],[178,3936],{"x":180,"y":180,"href":3190},[178,3938],{"x":3193,"y":180,"href":1484},[178,3940],{"x":3196,"y":180,"href":3197},[178,3942],{"x":3200,"y":180,"href":1636},[178,3944],{"x":3203,"y":180,"href":1492},[178,3946],{"x":3206,"y":180,"href":3207},[178,3948],{"x":3210,"y":180,"href":3211},[138,3950,3951],{"transform":184},[186,3952,3953],{"x":180,"y":188,"width":3216,"height":190},[192,3954,3220],{"className":3955,"style":196},[195],[138,3957,3959],{"transform":3958,"dataTid":1505},"translate(39.783,14.836)",[138,3960,3962,3964],{"dataTid":1508,"className":3961,"fill":175,"transform":176},[174],[178,3963],{"x":180,"y":180,"href":1512},[138,3965,3966],{"transform":184},[186,3967,3968],{"x":180,"y":188,"width":1517,"height":190},[192,3969,238],{"className":3970,"style":196},[195],[138,3972,3975],{"transform":3973,"dataTid":3974},"translate(44.062,14.836)","p5XqcLQe5VikdD\u002F0gU30OZs",[138,3976,3979,3982],{"dataTid":3977,"className":3978,"fill":175,"transform":176},"g5XqcLQe5VikdD\u002F0gU30OZs",[174],[178,3980],{"x":180,"y":180,"href":3981},"#gl3Hi1SAL",[138,3983,3984],{"transform":184},[186,3985,3987],{"x":180,"y":188,"width":3986,"height":190},"30.94",[192,3988,3990],{"className":3989,"style":196},[195],"𝑧",[138,3992,3994],{"transform":3993,"dataTid":1539},"translate(49.507,14.836)",[138,3995,3997,3999],{"dataTid":1542,"className":3996,"fill":175,"transform":176},[174],[178,3998],{"x":180,"y":180,"href":1546},[138,4000,4001],{"transform":184},[186,4002,4003],{"x":180,"y":188,"width":1517,"height":190},[192,4004,472],{"className":4005,"style":196},[195],[138,4007,4010],{"transform":4008,"dataTid":4009},"translate(53.786,17.553)","ppug+2aEXhoIic4bAxCHREs",[138,4011,4014,4017],{"dataTid":4012,"className":4013,"fill":175,"transform":206},"gpug+2aEXhoIic4bAxCHREs",[174],[178,4015],{"x":180,"y":180,"href":4016},"#gl3Hi1dQT",[138,4018,4019],{"transform":184},[186,4020,4022],{"x":180,"y":188,"width":4021,"height":190},"25.25",[192,4023,4025],{"className":4024,"style":196},[195],"𝑖",[138,4027,4029],{"transform":4028,"dataTid":1442},"translate(60.569,14.836)",[138,4030,4032,4034],{"dataTid":1445,"className":4031,"fill":175,"transform":176},[174],[178,4033],{"x":180,"y":180,"href":1449},[138,4035,4036],{"transform":184},[186,4037,4038],{"x":180,"y":188,"width":1454,"height":190},[192,4039,946],{"className":4040,"style":196},[195],[138,4042,4045],{"transform":4043,"dataTid":4044},"translate(72.182,0.000)","pcE+2OfY9QmstWWdFj8MCHc",[138,4046,4049,4064,4083,4103,4122,4142,4269],{"dataTid":4047,"className":4048},"gcE+2OfY9QmstWWdFj8MCHc",[150],[138,4050,4052],{"transform":4051,"dataTid":3781},"translate(10.102,7.389)",[138,4053,4055,4057],{"dataTid":3784,"className":4054,"fill":175,"transform":176},[174],[178,4056],{"x":180,"y":180,"href":3788},[138,4058,4059],{"transform":184},[186,4060,4061],{"x":180,"y":188,"width":3793,"height":190},[192,4062,3797],{"className":4063,"style":196},[195],[138,4065,4068],{"transform":4066,"dataTid":4067},"translate(15.228,3.396)","pT3U2V8ZWLSMvbKUGZx0sFM",[138,4069,4072,4075],{"dataTid":4070,"className":4071,"fill":175,"transform":206},"gT3U2V8ZWLSMvbKUGZx0sFM",[174],[178,4073],{"x":180,"y":180,"href":4074},"#gl3Hi1eUT",[138,4076,4077],{"transform":184},[186,4078,4080],{"x":180,"y":188,"width":4079,"height":190},"34.31",[192,4081,3990],{"className":4082,"style":196},[195],[138,4084,4087],{"transform":4085,"dataTid":4086},"translate(19.432,5.298)","pqZ6Uz1W6KYs7XnhEmYF0jI",[138,4088,4092,4095],{"dataTid":4089,"className":4090,"fill":175,"transform":4091},"gqZ6Uz1W6KYs7XnhEmYF0jI",[174],"scale(0.0055,-0.0055)",[178,4093],{"x":180,"y":180,"href":4094},"#gl3Hi1Z4T",[138,4096,4097],{"transform":184},[186,4098,4100],{"x":180,"y":188,"width":4099,"height":190},"33.38",[192,4101,4025],{"className":4102,"style":196},[195],[138,4104,4107],{"transform":4105,"dataTid":4106},"translate(22.801,3.396)","pgssd\u002FwcxXeQXfwOpASaeOU",[138,4108,4111,4114],{"dataTid":4109,"className":4110,"fill":175,"transform":206},"ggssd\u002FwcxXeQXfwOpASaeOU",[174],[178,4112],{"x":180,"y":180,"href":4113},"#gl3Hi1aAb",[138,4115,4116],{"transform":184},[186,4117,4118],{"x":180,"y":188,"width":294,"height":190},[192,4119,4121],{"className":4120,"style":196},[195],"−",[138,4123,4126],{"transform":4124,"dataTid":4125},"translate(28.791,3.396)","pHsTZVcmqv2o9AV52+BZWAc",[138,4127,4130,4133],{"dataTid":4128,"className":4129,"fill":175,"transform":206},"gHsTZVcmqv2o9AV52+BZWAc",[174],[178,4131],{"x":180,"y":180,"href":4132},"#gl3Hi1dgT",[138,4134,4135],{"transform":184},[186,4136,4138],{"x":180,"y":188,"width":4137,"height":190},"63.38",[192,4139,4141],{"className":4140,"style":196},[195],"𝑚",[138,4143,4146],{"transform":4144,"dataTid":4145},"translate(1.100,14.132)","pr41xZ0z5zl0b3VixMn3j3g",[138,4147,4150,4170,4190,4205,4220,4239,4254],{"dataTid":4148,"className":4149},"gr41xZ0z5zl0b3VixMn3j3g",[150],[138,4151,4154],{"transform":4152,"dataTid":4153},"translate(0.000,8.250)","pXUFdkI8lsxgunoXLbLLAHE",[138,4155,4158,4161],{"dataTid":4156,"className":4157,"fill":175,"transform":176},"gXUFdkI8lsxgunoXLbLLAHE",[174],[178,4159],{"x":180,"y":180,"href":4160},"#gl3Hi1dQD",[138,4162,4163],{"transform":184},[186,4164,4166],{"x":180,"y":188,"width":4165,"height":190},"66.00",[192,4167,4169],{"className":4168,"style":196},[195],"∑",[138,4171,4174],{"transform":4172,"dataTid":4173},"translate(11.616,13.200)","psQcoFQ3Bd2sZBq3KQKpm9M",[138,4175,4178,4181],{"dataTid":4176,"className":4177,"fill":175,"transform":206},"gsQcoFQ3Bd2sZBq3KQKpm9M",[174],[178,4179],{"x":180,"y":180,"href":4180},"#gl3Hi1dUT",[138,4182,4183],{"transform":184},[186,4184,4186],{"x":180,"y":188,"width":4185,"height":190},"29.56",[192,4187,4189],{"className":4188,"style":196},[195],"𝑗",[138,4191,4193],{"transform":4192,"dataTid":3781},"translate(17.707,8.250)",[138,4194,4196,4198],{"dataTid":3784,"className":4195,"fill":175,"transform":176},[174],[178,4197],{"x":180,"y":180,"href":3788},[138,4199,4200],{"transform":184},[186,4201,4202],{"x":180,"y":188,"width":3793,"height":190},[192,4203,3797],{"className":4204,"style":196},[195],[138,4206,4208],{"transform":4207,"dataTid":4067},"translate(22.833,4.033)",[138,4209,4211,4213],{"dataTid":4070,"className":4210,"fill":175,"transform":206},[174],[178,4212],{"x":180,"y":180,"href":4074},[138,4214,4215],{"transform":184},[186,4216,4217],{"x":180,"y":188,"width":4079,"height":190},[192,4218,3990],{"className":4219,"style":196},[195],[138,4221,4224],{"transform":4222,"dataTid":4223},"translate(27.038,5.935)","pXIjEPaXVgWEPPr5fIVn6GI",[138,4225,4228,4231],{"dataTid":4226,"className":4227,"fill":175,"transform":4091},"gXIjEPaXVgWEPPr5fIVn6GI",[174],[178,4229],{"x":180,"y":180,"href":4230},"#gl3Hi1Z8T",[138,4232,4233],{"transform":184},[186,4234,4236],{"x":180,"y":188,"width":4235,"height":190},"36.75",[192,4237,4189],{"className":4238,"style":196},[195],[138,4240,4242],{"transform":4241,"dataTid":4106},"translate(30.703,4.033)",[138,4243,4245,4247],{"dataTid":4109,"className":4244,"fill":175,"transform":206},[174],[178,4246],{"x":180,"y":180,"href":4113},[138,4248,4249],{"transform":184},[186,4250,4251],{"x":180,"y":188,"width":294,"height":190},[192,4252,4121],{"className":4253,"style":196},[195],[138,4255,4257],{"transform":4256,"dataTid":4125},"translate(36.693,4.033)",[138,4258,4260,4262],{"dataTid":4128,"className":4259,"fill":175,"transform":206},[174],[178,4261],{"x":180,"y":180,"href":4132},[138,4263,4264],{"transform":184},[186,4265,4266],{"x":180,"y":188,"width":4137,"height":190},[192,4267,4141],{"className":4268,"style":196},[195],[138,4270,4273],{"transform":4271,"dataTid":4272},"translate(1.100,12.086)","pwvAGtMVQb5I6Re15GNSW68",[138,4274,4276],{"dataTid":4275},"gwvAGtMVQb5I6Re15GNSW68",[75,4277],{"className":4278,"d":4279,"strokeWidth":3324,"strokeMiterLimit":970,"fillRule":3325,"fill":3326,"stroke":175},[3322],"M 0 0 L 45.117233 0 ",[138,4281,4284],{"transform":4282,"dataTid":4283},"translate(130.499,14.836)","phDSRiqSYRWEg4fwXbbJ784",[138,4285,4288,4291],{"dataTid":4286,"className":4287,"fill":175,"transform":176},"ghDSRiqSYRWEg4fwXbbJ784",[174],[178,4289],{"x":180,"y":180,"href":4290},"#gl3Hi1RML",[138,4292,4293],{"transform":184},[186,4294,4296],{"x":180,"y":188,"width":4295,"height":190},"54.88",[192,4297,4141],{"className":4298,"style":196},[195],[138,4300,4302],{"transform":4301,"dataTid":1442},"translate(143.213,14.836)",[138,4303,4305,4307],{"dataTid":1445,"className":4304,"fill":175,"transform":176},[174],[178,4306],{"x":180,"y":180,"href":1449},[138,4308,4309],{"transform":184},[186,4310,4311],{"x":180,"y":188,"width":1454,"height":190},[192,4312,946],{"className":4313,"style":196},[195],[138,4315,4318],{"transform":4316,"dataTid":4317},"translate(154.827,14.836)","p2YxGTkVM144RI7vMrsT1kw",[138,4319,4322,4324,4327,4330],{"dataTid":4320,"className":4321,"fill":175,"transform":176},"g2YxGTkVM144RI7vMrsT1kw",[174],[178,4323],{"x":180,"y":180,"href":1492},[178,4325],{"x":4326,"y":180,"href":3207},"833",[178,4328],{"x":4329,"y":180,"href":3211},"1333",[138,4331,4332],{"transform":184},[186,4333,4335],{"x":180,"y":188,"width":4334,"height":190},"116.31",[192,4336,4338],{"className":4337,"style":196},[195],"max",[138,4340,4342],{"transform":4341,"dataTid":4173},"translate(163.241,21.899)",[138,4343,4345,4347],{"dataTid":4176,"className":4344,"fill":175,"transform":206},[174],[178,4346],{"x":180,"y":180,"href":4180},[138,4348,4349],{"transform":184},[186,4350,4351],{"x":180,"y":188,"width":4185,"height":190},[192,4352,4189],{"className":4353,"style":196},[195],[138,4355,4357],{"transform":4356,"dataTid":3974},"translate(177.131,14.836)",[138,4358,4360,4362],{"dataTid":3977,"className":4359,"fill":175,"transform":176},[174],[178,4361],{"x":180,"y":180,"href":3981},[138,4363,4364],{"transform":184},[186,4365,4366],{"x":180,"y":188,"width":3986,"height":190},[192,4367,3990],{"className":4368,"style":196},[195],[138,4370,4372],{"transform":4371,"dataTid":4173},"translate(182.246,17.553)",[138,4373,4375,4377],{"dataTid":4176,"className":4374,"fill":175,"transform":206},[174],[178,4376],{"x":180,"y":180,"href":4180},[138,4378,4379],{"transform":184},[186,4380,4381],{"x":180,"y":188,"width":4185,"height":190},[192,4382,4189],{"className":4383,"style":196},[195],[138,4385,4386],{"transform":153,"dataTid":475},[138,4387],{"dataTid":478,"className":4388,"dataHint":15},[480],[138,4390,4391],{"transform":153,"dataTid":475},[138,4392],{"dataTid":478,"className":4393,"dataHint":15},[480],[11,4395,4396,4397,4448],{},"We first subtract the largest score from every score. The final weights stay the same, but the exponentials stay at 1 or below, which helps prevent overflow. A masked score becomes zero after the exponential because ",[51,4398,4400,4402,4408,4411,4413],{"style":491,"className":4399,"viewBox":493,"width":494,"height":495,"dataWidth":496,"dataHeight":497,"xmlns":61,"xmlnsXLink":62,"xmlns:h5":63},[55],[65,4401,68],{"type":67},[70,4403,4405],{"className":4404},[73],[75,4406],{"id":3749,"className":4407,"d":3751},[79],[70,4409],{"className":4410},[134],[65,4412],{"type":67},[138,4414,4416],{"className":4415,"transform":142,"dataTid":3760,"dataPageWidth":517,"dataPageHeight":518},[141],[138,4417,4419],{"dataTid":3763,"className":4418},[150],[138,4420,4421],{"transform":153,"dataTid":3767},[138,4422,4424,4443],{"dataTid":3770,"className":4423},[150],[138,4425,4426],{"transform":153,"dataTid":3774},[138,4427,4429],{"dataTid":3777,"className":4428},[150],[138,4430,4431],{"transform":539,"dataTid":3781},[138,4432,4434,4436],{"dataTid":3784,"className":4433,"fill":175,"transform":176},[174],[178,4435],{"x":180,"y":180,"href":3788},[138,4437,4438],{"transform":184},[186,4439,4440],{"x":180,"y":188,"width":3793,"height":190},[192,4441,3797],{"className":4442,"style":196},[195],[138,4444,4445],{"transform":153,"dataTid":475},[138,4446],{"dataTid":478,"className":4447,"dataHint":15},[480]," to the power of negative infinity is zero. The Rust function follows these steps directly.",[898,4450,4452],{"className":900,"code":4451,"language":902,"meta":903,"style":903},"fn softmax(scores: &[f32]) -> Vec\u003Cf32> {\n    let max = scores.iter().copied().fold(f32::NEG_INFINITY, f32::max);\n    let exps: Vec\u003Cf32> = scores.iter().map(|&s| (s - max).exp()).collect();\n    let sum: f32 = exps.iter().sum();\n    exps.iter().map(|&e| e \u002F sum).collect()\n}\n",[868,4453,4454,4493,4542,4607,4637,4677],{"__ignoreMap":903},[907,4455,4456,4459,4462,4465,4468,4471,4474,4476,4479,4482,4485,4488,4490],{"class":909,"line":910},[907,4457,4458],{"class":913},"fn",[907,4460,4461],{"class":916}," softmax",[907,4463,4464],{"class":926},"(scores",[907,4466,4467],{"class":913},":",[907,4469,4470],{"class":913}," &",[907,4472,4473],{"class":926},"[",[907,4475,3711],{"class":916},[907,4477,4478],{"class":926},"]) ",[907,4480,4481],{"class":913},"->",[907,4483,4484],{"class":916}," Vec",[907,4486,4487],{"class":926},"\u003C",[907,4489,3711],{"class":916},[907,4491,4492],{"class":926},"> {\n",[907,4494,4495,4498,4501,4503,4506,4508,4511,4513,4515,4518,4520,4522,4525,4527,4529,4531,4533,4535,4537,4539],{"class":909,"line":930},[907,4496,4497],{"class":913},"    let",[907,4499,4500],{"class":926}," max ",[907,4502,946],{"class":913},[907,4504,4505],{"class":926}," scores",[907,4507,1072],{"class":913},[907,4509,4510],{"class":916},"iter",[907,4512,1149],{"class":926},[907,4514,1072],{"class":913},[907,4516,4517],{"class":916},"copied",[907,4519,1149],{"class":926},[907,4521,1072],{"class":913},[907,4523,4524],{"class":916},"fold",[907,4526,238],{"class":926},[907,4528,3711],{"class":916},[907,4530,920],{"class":913},[907,4532,3727],{"class":1007},[907,4534,1195],{"class":926},[907,4536,3711],{"class":916},[907,4538,920],{"class":913},[907,4540,4541],{"class":926},"max);\n",[907,4543,4544,4546,4549,4551,4553,4555,4557,4560,4562,4564,4566,4568,4570,4572,4574,4576,4579,4582,4584,4587,4590,4593,4595,4598,4600,4602,4605],{"class":909,"line":937},[907,4545,4497],{"class":913},[907,4547,4548],{"class":926}," exps",[907,4550,4467],{"class":913},[907,4552,4484],{"class":916},[907,4554,4487],{"class":926},[907,4556,3711],{"class":916},[907,4558,4559],{"class":926},"> ",[907,4561,946],{"class":913},[907,4563,4505],{"class":926},[907,4565,1072],{"class":913},[907,4567,4510],{"class":916},[907,4569,1149],{"class":926},[907,4571,1072],{"class":913},[907,4573,3675],{"class":916},[907,4575,238],{"class":926},[907,4577,4578],{"class":913},"|&",[907,4580,4581],{"class":926},"s",[907,4583,338],{"class":913},[907,4585,4586],{"class":926}," (s ",[907,4588,4589],{"class":913},"-",[907,4591,4592],{"class":926}," max)",[907,4594,1072],{"class":913},[907,4596,4597],{"class":916},"exp",[907,4599,2429],{"class":926},[907,4601,1072],{"class":913},[907,4603,4604],{"class":916},"collect",[907,4606,3008],{"class":926},[907,4608,4609,4611,4614,4616,4619,4622,4624,4626,4628,4630,4632,4635],{"class":909,"line":970},[907,4610,4497],{"class":913},[907,4612,4613],{"class":926}," sum",[907,4615,4467],{"class":913},[907,4617,4618],{"class":916}," f32",[907,4620,4621],{"class":913}," =",[907,4623,4548],{"class":926},[907,4625,1072],{"class":913},[907,4627,4510],{"class":916},[907,4629,1149],{"class":926},[907,4631,1072],{"class":913},[907,4633,4634],{"class":916},"sum",[907,4636,3008],{"class":926},[907,4638,4639,4642,4644,4646,4648,4650,4652,4654,4656,4659,4661,4664,4667,4670,4672,4674],{"class":909,"line":985},[907,4640,4641],{"class":926},"    exps",[907,4643,1072],{"class":913},[907,4645,4510],{"class":916},[907,4647,1149],{"class":926},[907,4649,1072],{"class":913},[907,4651,3675],{"class":916},[907,4653,238],{"class":926},[907,4655,4578],{"class":913},[907,4657,4658],{"class":926},"e",[907,4660,338],{"class":913},[907,4662,4663],{"class":926}," e ",[907,4665,4666],{"class":913},"\u002F",[907,4668,4669],{"class":926}," sum)",[907,4671,1072],{"class":913},[907,4673,4604],{"class":916},[907,4675,4676],{"class":926},"()\n",[907,4678,4679],{"class":909,"line":1001},[907,4680,4681],{"class":926},"}\n",[11,4683,4684],{},"We ran this function on all 47 rows of the captured scores, with the causal mask applied. The largest difference from the reference F32 softmax was about 0.00000042.",[11,4686,4687],{},[850,4688],{"alt":4689,"src":4690},"Scores and softmax weights for the last prompt position in layer 15, head 1, across all 47 key positions","\u002Fassets\u002Fblog\u002Ffrom-safetensors-to-tokens-executing-a-transformer-step-by-step\u002Fsoftmax.svg",[11,4692,4693,4694,4697],{},"The last row belongs to prompt position 46. Its scaled scores range from about -6.4 to 2.8. In this head, ",[868,4695,4696],{},"\u003C|begin_of_text|>"," receives 77.2 percent of the attention weight. \" word\" gets 4.4 percent, \" France\" 3.1 percent and \" capital\" 2.3 percent.",[11,4699,4700],{},"These numbers describe how this head mixes the value vectors. They do not tell us which word caused the answer. We are looking at one head in one layer, with more operations still to come.",[11,4702,4703],{},"Foundry builds softmax from the same steps, written as tensor operations that run on the GPU.",[898,4705,4707],{"className":900,"code":4706,"language":902,"meta":903,"style":903},"let x_f32 = x.to_dtype(device, DType::F32)?;\nlet row_max = x_f32.amax_keepdim(device, dim)?;\nlet exps = x_f32.sub(device, &row_max)?.exp(device)?;\nlet row_sum = exps.sum_keepdim(device, dim)?;\nexps.div(device, &row_sum)?.to_dtype(device, x.dtype())\n",[868,4708,4709,4737,4760,4793,4814],{"__ignoreMap":903},[907,4710,4711,4713,4715,4717,4719,4721,4723,4725,4727,4729,4731,4733,4735],{"class":909,"line":910},[907,4712,940],{"class":913},[907,4714,2265],{"class":926},[907,4716,946],{"class":913},[907,4718,2270],{"class":926},[907,4720,1072],{"class":913},[907,4722,2275],{"class":916},[907,4724,1190],{"class":926},[907,4726,2280],{"class":916},[907,4728,920],{"class":913},[907,4730,2285],{"class":916},[907,4732,472],{"class":926},[907,4734,965],{"class":913},[907,4736,927],{"class":926},[907,4738,4739,4741,4744,4746,4749,4751,4754,4756,4758],{"class":909,"line":930},[907,4740,940],{"class":913},[907,4742,4743],{"class":926}," row_max ",[907,4745,946],{"class":913},[907,4747,4748],{"class":926}," x_f32",[907,4750,1072],{"class":913},[907,4752,4753],{"class":916},"amax_keepdim",[907,4755,2331],{"class":926},[907,4757,965],{"class":913},[907,4759,927],{"class":926},[907,4761,4762,4764,4767,4769,4771,4773,4776,4778,4780,4783,4785,4787,4789,4791],{"class":909,"line":937},[907,4763,940],{"class":913},[907,4765,4766],{"class":926}," exps ",[907,4768,946],{"class":913},[907,4770,4748],{"class":926},[907,4772,1072],{"class":913},[907,4774,4775],{"class":916},"sub",[907,4777,1190],{"class":926},[907,4779,1198],{"class":913},[907,4781,4782],{"class":926},"row_max)",[907,4784,2416],{"class":913},[907,4786,4597],{"class":916},[907,4788,2382],{"class":926},[907,4790,965],{"class":913},[907,4792,927],{"class":926},[907,4794,4795,4797,4800,4802,4804,4806,4808,4810,4812],{"class":909,"line":970},[907,4796,940],{"class":913},[907,4798,4799],{"class":926}," row_sum ",[907,4801,946],{"class":913},[907,4803,4548],{"class":926},[907,4805,1072],{"class":913},[907,4807,2328],{"class":916},[907,4809,2331],{"class":926},[907,4811,965],{"class":913},[907,4813,927],{"class":926},[907,4815,4816,4819,4821,4824,4826,4828,4831,4833,4835,4837,4839,4841],{"class":909,"line":985},[907,4817,4818],{"class":926},"exps",[907,4820,1072],{"class":913},[907,4822,4823],{"class":916},"div",[907,4825,1190],{"class":926},[907,4827,1198],{"class":913},[907,4829,4830],{"class":926},"row_sum)",[907,4832,2416],{"class":913},[907,4834,2275],{"class":916},[907,4836,2421],{"class":926},[907,4838,1072],{"class":913},[907,4840,2426],{"class":916},[907,4842,4843],{"class":926},"())\n",[11,4845,4846],{},"In this implementation, the tensor operations run separately on the GPU and write their results to memory. A fused implementation combines steps in one GPU kernel, reducing the trips to memory.",[11,4848,4849,4850,4852],{},"The second matrix multiplication uses the weights to mix the values. The last position gets a vector of 128 numbers. The value vector of ",[868,4851,4696],{}," is multiplied by about 0.772, and each other value vector is multiplied by its own weight. These scaled vectors are then added together.",[898,4854,4856],{"className":900,"code":4855,"language":902,"meta":903,"style":903},"softmax(device, &scores, scores.shape().last_dim()?)?\n    .to_dtype(device, query.dtype())?\n    .reshape(Shape::try_from([batch, kv_heads, rows, kv_length])?)?\n    .matmul(device, value)?\n",[868,4857,4858,4888,4905,4928],{"__ignoreMap":903},[907,4859,4860,4862,4864,4866,4869,4871,4873,4875,4877,4880,4882,4884,4886],{"class":909,"line":910},[907,4861,3220],{"class":916},[907,4863,1190],{"class":926},[907,4865,1198],{"class":913},[907,4867,4868],{"class":926},"scores, scores",[907,4870,1072],{"class":913},[907,4872,1146],{"class":916},[907,4874,1149],{"class":926},[907,4876,1072],{"class":913},[907,4878,4879],{"class":916},"last_dim",[907,4881,1149],{"class":926},[907,4883,965],{"class":913},[907,4885,472],{"class":926},[907,4887,2321],{"class":913},[907,4889,4890,4892,4894,4897,4899,4901,4903],{"class":909,"line":930},[907,4891,2308],{"class":913},[907,4893,2275],{"class":916},[907,4895,4896],{"class":926},"(device, query",[907,4898,1072],{"class":913},[907,4900,2426],{"class":916},[907,4902,2429],{"class":926},[907,4904,2321],{"class":913},[907,4906,4907,4909,4911,4913,4915,4917,4919,4922,4924,4926],{"class":909,"line":937},[907,4908,2308],{"class":913},[907,4910,1128],{"class":916},[907,4912,238],{"class":926},[907,4914,1133],{"class":916},[907,4916,920],{"class":913},[907,4918,1138],{"class":916},[907,4920,4921],{"class":926},"([batch, kv_heads, rows, kv_length])",[907,4923,965],{"class":913},[907,4925,472],{"class":926},[907,4927,2321],{"class":913},[907,4929,4930,4932,4934,4937],{"class":909,"line":970},[907,4931,2308],{"class":913},[907,4933,3568],{"class":916},[907,4935,4936],{"class":926},"(device, value)",[907,4938,2321],{"class":913},[11,4940,4941],{},"The 24 head outputs are joined back into 3072 numbers, and the output projection mixes them. That result is added to the residual stream, which closes the attention half of the block.",[43,4943,4945],{"id":4944},"the-mlp-and-the-rest-of-the-stack","The MLP and the rest of the stack",[11,4947,4948],{},"The MLP works on one position at a time. Two projections each turn its 3072 numbers into 8192 numbers. We apply SiLU, an activation function, to one result and multiply it by the other. A final projection brings the result back to 3072 numbers. This is called a gated MLP, or SwiGLU.",[898,4950,4952],{"className":900,"code":4951,"language":902,"meta":903,"style":903},"let gate = self.gate_proj.forward(device, x)?;\nlet up = self.up_proj.forward(device, x)?;\nlet hidden = self.act_fn.forward(device, &gate)?.mul(device, &up)?;\nself.down_proj.forward(device, &hidden)\n",[868,4953,4954,4981,5007,5049],{"__ignoreMap":903},[907,4955,4956,4958,4961,4963,4965,4967,4970,4972,4974,4977,4979],{"class":909,"line":910},[907,4957,940],{"class":913},[907,4959,4960],{"class":926}," gate ",[907,4962,946],{"class":913},[907,4964,1177],{"class":1007},[907,4966,1072],{"class":913},[907,4968,4969],{"class":926},"gate_proj",[907,4971,1072],{"class":913},[907,4973,2107],{"class":916},[907,4975,4976],{"class":926},"(device, x)",[907,4978,965],{"class":913},[907,4980,927],{"class":926},[907,4982,4983,4985,4988,4990,4992,4994,4997,4999,5001,5003,5005],{"class":909,"line":930},[907,4984,940],{"class":913},[907,4986,4987],{"class":926}," up ",[907,4989,946],{"class":913},[907,4991,1177],{"class":1007},[907,4993,1072],{"class":913},[907,4995,4996],{"class":926},"up_proj",[907,4998,1072],{"class":913},[907,5000,2107],{"class":916},[907,5002,4976],{"class":926},[907,5004,965],{"class":913},[907,5006,927],{"class":926},[907,5008,5009,5011,5014,5016,5018,5020,5023,5025,5027,5029,5031,5034,5036,5038,5040,5042,5045,5047],{"class":909,"line":937},[907,5010,940],{"class":913},[907,5012,5013],{"class":926}," hidden ",[907,5015,946],{"class":913},[907,5017,1177],{"class":1007},[907,5019,1072],{"class":913},[907,5021,5022],{"class":926},"act_fn",[907,5024,1072],{"class":913},[907,5026,2107],{"class":916},[907,5028,1190],{"class":926},[907,5030,1198],{"class":913},[907,5032,5033],{"class":926},"gate)",[907,5035,2416],{"class":913},[907,5037,2311],{"class":916},[907,5039,1190],{"class":926},[907,5041,1198],{"class":913},[907,5043,5044],{"class":926},"up)",[907,5046,965],{"class":913},[907,5048,927],{"class":926},[907,5050,5051,5053,5055,5058,5060,5062,5064,5066],{"class":909,"line":970},[907,5052,2346],{"class":1007},[907,5054,1072],{"class":913},[907,5056,5057],{"class":926},"down_proj",[907,5059,1072],{"class":913},[907,5061,2107],{"class":916},[907,5063,1190],{"class":926},[907,5065,1198],{"class":913},[907,5067,5068],{"class":926},"hidden)\n",[11,5070,5071],{},"Llama 3.2 3B stacks 28 of these blocks. They all have the same structure, each with its own weights, and the output of one block is the input of the next. Foundry passes the output through the list of layers, then applies a final RMSNorm.",[898,5073,5075],{"className":900,"code":5074,"language":902,"meta":903,"style":903},"let hidden_states = self.layers.iter().try_fold(inputs_embeds, |hidden_states, layer| {\n    layer.forward(device, &hidden_states, [&cos, &sin], &attention_mask, past_key_values, past_seen_tokens)\n})?;\nself.norm.forward(device, &hidden_states)\n",[868,5076,5077,5115,5146,5155],{"__ignoreMap":903},[907,5078,5079,5081,5083,5085,5087,5089,5092,5094,5096,5098,5100,5103,5106,5108,5111,5113],{"class":909,"line":910},[907,5080,940],{"class":913},[907,5082,2152],{"class":926},[907,5084,946],{"class":913},[907,5086,1177],{"class":1007},[907,5088,1072],{"class":913},[907,5090,5091],{"class":926},"layers",[907,5093,1072],{"class":913},[907,5095,4510],{"class":916},[907,5097,1149],{"class":926},[907,5099,1072],{"class":913},[907,5101,5102],{"class":916},"try_fold",[907,5104,5105],{"class":926},"(inputs_embeds, ",[907,5107,338],{"class":913},[907,5109,5110],{"class":926},"hidden_states, layer",[907,5112,338],{"class":913},[907,5114,3691],{"class":926},[907,5116,5117,5120,5122,5124,5126,5128,5131,5133,5136,5138,5141,5143],{"class":909,"line":930},[907,5118,5119],{"class":926},"    layer",[907,5121,1072],{"class":913},[907,5123,2107],{"class":916},[907,5125,1190],{"class":926},[907,5127,1198],{"class":913},[907,5129,5130],{"class":926},"hidden_states, [",[907,5132,1198],{"class":913},[907,5134,5135],{"class":926},"cos, ",[907,5137,1198],{"class":913},[907,5139,5140],{"class":926},"sin], ",[907,5142,1198],{"class":913},[907,5144,5145],{"class":926},"attention_mask, past_key_values, past_seen_tokens)\n",[907,5147,5148,5151,5153],{"class":909,"line":937},[907,5149,5150],{"class":926},"})",[907,5152,965],{"class":913},[907,5154,927],{"class":926},[907,5156,5157,5159,5161,5164,5166,5168,5170,5172],{"class":909,"line":970},[907,5158,2346],{"class":1007},[907,5160,1072],{"class":913},[907,5162,5163],{"class":926},"norm",[907,5165,1072],{"class":913},[907,5167,2107],{"class":916},[907,5169,1190],{"class":926},[907,5171,1198],{"class":913},[907,5173,5174],{"class":926},"hidden_states)\n",[43,5176,5178],{"id":5177},"picking-the-next-token","Picking the next token",[11,5180,5181],{},"After the last block and the final RMSNorm, each position still holds 3072 numbers. Only the last position matters for the next token, so Foundry drops the others. The LM head multiplies that last vector by a matrix of 128,256 rows and returns one score per vocabulary entry, the logits. Llama 3.2 reuses the embedding table for this, so Foundry wires the two together when it loads the model. The clone creates another handle to the same weight tensor.",[898,5183,5185],{"className":900,"code":5184,"language":902,"meta":903,"style":903},"let lm_head = Linear::new(model.embed_tokens.weight().clone());\n",[868,5186,5187],{"__ignoreMap":903},[907,5188,5189,5191,5194,5196,5199,5201,5204,5207,5209,5212,5214,5216,5218,5220,5223],{"class":909,"line":910},[907,5190,940],{"class":913},[907,5192,5193],{"class":926}," lm_head ",[907,5195,946],{"class":913},[907,5197,5198],{"class":916}," Linear",[907,5200,920],{"class":913},[907,5202,5203],{"class":916},"new",[907,5205,5206],{"class":926},"(model",[907,5208,1072],{"class":913},[907,5210,5211],{"class":926},"embed_tokens",[907,5213,1072],{"class":913},[907,5215,1182],{"class":916},[907,5217,1149],{"class":926},[907,5219,1072],{"class":913},[907,5221,5222],{"class":916},"clone",[907,5224,5225],{"class":926},"());\n",[11,5227,5228],{},"To see the probabilities, we apply softmax again, this time over 128,256 entries instead of 47. The probabilities below come from the Transformers logits. We also checked those logits with the standalone Rust function. Its largest probability difference was about 0.0000065.",[898,5230,5232],{"className":900,"code":5231,"language":902,"meta":903,"style":903},"let probs = softmax(&logits);\nlet (id, p) = probs.iter().enumerate().fold((0, 0f32), |best, (i, &p)| if p > best.1 { (i, p) } else { best });\n\u002F\u002F id 60704, p about 0.99967, decodes to \"Paris\"\n",[868,5233,5234,5252,5331],{"__ignoreMap":903},[907,5235,5236,5238,5241,5243,5245,5247,5249],{"class":909,"line":910},[907,5237,940],{"class":913},[907,5239,5240],{"class":926}," probs ",[907,5242,946],{"class":913},[907,5244,4461],{"class":916},[907,5246,238],{"class":926},[907,5248,1198],{"class":913},[907,5250,5251],{"class":926},"logits);\n",[907,5253,5254,5256,5259,5261,5264,5266,5268,5270,5272,5275,5277,5279,5281,5284,5286,5288,5290,5292,5295,5297,5300,5302,5305,5307,5310,5313,5316,5319,5321,5323,5326,5328],{"class":909,"line":930},[907,5255,940],{"class":913},[907,5257,5258],{"class":926}," (id, p) ",[907,5260,946],{"class":913},[907,5262,5263],{"class":926}," probs",[907,5265,1072],{"class":913},[907,5267,4510],{"class":916},[907,5269,1149],{"class":926},[907,5271,1072],{"class":913},[907,5273,5274],{"class":916},"enumerate",[907,5276,1149],{"class":926},[907,5278,1072],{"class":913},[907,5280,4524],{"class":916},[907,5282,5283],{"class":926},"((",[907,5285,180],{"class":1007},[907,5287,1195],{"class":926},[907,5289,180],{"class":1007},[907,5291,3711],{"class":916},[907,5293,5294],{"class":926},"), ",[907,5296,338],{"class":913},[907,5298,5299],{"class":926},"best, (i, ",[907,5301,1198],{"class":913},[907,5303,5304],{"class":926},"p)",[907,5306,338],{"class":913},[907,5308,5309],{"class":913}," if",[907,5311,5312],{"class":926}," p ",[907,5314,5315],{"class":913},">",[907,5317,5318],{"class":926}," best",[907,5320,1072],{"class":913},[907,5322,318],{"class":1007},[907,5324,5325],{"class":926}," { (i, p) } ",[907,5327,3717],{"class":913},[907,5329,5330],{"class":926}," { best });\n",[907,5332,5333],{"class":909,"line":937},[907,5334,5335],{"class":1093},"\u002F\u002F id 60704, p about 0.99967, decodes to \"Paris\"\n",[11,5337,5338],{},"Token 60704, \"Paris\" with no leading space, gets 99.966 percent in the reference run. The next candidates are \"Par\" at about 0.023 percent and \" Paris\" with a leading space at about 0.0024 percent. These are three different vocabulary entries. The probability describes this particular prompt and model run. It does not measure whether an answer is true.",[11,5340,5341],{},[850,5342],{"alt":5343,"src":5344},"Probabilities of the first answer token, led by \"Paris\" at 99.966 percent, with the rest of the vocabulary grouped","\u002Fassets\u002Fblog\u002Ffrom-safetensors-to-tokens-executing-a-transformer-step-by-step\u002Fnext-token.svg",[11,5346,5347],{},"Greedy decoding takes the highest logit directly. It does not need softmax because softmax keeps the scores in the same order. Sampling draws from the probabilities instead, often after applying a temperature to make the distribution sharper or flatter. Foundry uses greedy decoding and finds the top score on the GPU, so only one integer per sequence goes back to the CPU.",[898,5349,5351],{"className":900,"code":5350,"language":902,"meta":903,"style":903},"let bytes = logits\n    .argmax_keepdim(self.device, VOCAB_DIM)\n    .and_then(|token| token.to_bytes(self.device))\n    .map_err(|error| {\n        error!(message = \"Selecting the next token failed.\", %error);\n        GenerateError::NextToken\n    })?;\ndecode_token_ids(&bytes, self.batch_size)\n",[868,5352,5353,5365,5387,5420,5437,5458,5468,5477],{"__ignoreMap":903},[907,5354,5355,5357,5360,5362],{"class":909,"line":910},[907,5356,940],{"class":913},[907,5358,5359],{"class":926}," bytes ",[907,5361,946],{"class":913},[907,5363,5364],{"class":926}," logits\n",[907,5366,5367,5369,5372,5374,5376,5378,5381,5384],{"class":909,"line":930},[907,5368,2308],{"class":913},[907,5370,5371],{"class":916},"argmax_keepdim",[907,5373,238],{"class":926},[907,5375,2346],{"class":1007},[907,5377,1072],{"class":913},[907,5379,5380],{"class":926},"device, ",[907,5382,5383],{"class":1007},"VOCAB_DIM",[907,5385,5386],{"class":926},")\n",[907,5388,5389,5391,5394,5396,5398,5401,5403,5406,5408,5411,5413,5415,5417],{"class":909,"line":937},[907,5390,2308],{"class":913},[907,5392,5393],{"class":916},"and_then",[907,5395,238],{"class":926},[907,5397,338],{"class":913},[907,5399,5400],{"class":926},"token",[907,5402,338],{"class":913},[907,5404,5405],{"class":926}," token",[907,5407,1072],{"class":913},[907,5409,5410],{"class":916},"to_bytes",[907,5412,238],{"class":926},[907,5414,2346],{"class":1007},[907,5416,1072],{"class":913},[907,5418,5419],{"class":926},"device))\n",[907,5421,5422,5424,5426,5428,5430,5433,5435],{"class":909,"line":970},[907,5423,2308],{"class":913},[907,5425,2987],{"class":916},[907,5427,238],{"class":926},[907,5429,338],{"class":913},[907,5431,5432],{"class":926},"error",[907,5434,338],{"class":913},[907,5436,3691],{"class":926},[907,5438,5439,5442,5445,5447,5450,5452,5455],{"class":909,"line":985},[907,5440,5441],{"class":916},"        error!",[907,5443,5444],{"class":926},"(message ",[907,5446,946],{"class":913},[907,5448,5449],{"class":959}," \"Selecting the next token failed.\"",[907,5451,1195],{"class":926},[907,5453,5454],{"class":913},"%",[907,5456,5457],{"class":926},"error);\n",[907,5459,5460,5463,5465],{"class":909,"line":1001},[907,5461,5462],{"class":916},"        GenerateError",[907,5464,920],{"class":913},[907,5466,5467],{"class":916},"NextToken\n",[907,5469,5470,5473,5475],{"class":909,"line":1014},[907,5471,5472],{"class":926},"    })",[907,5474,965],{"class":913},[907,5476,927],{"class":926},[907,5478,5479,5482,5484,5486,5489,5491,5493],{"class":909,"line":1031},[907,5480,5481],{"class":916},"decode_token_ids",[907,5483,238],{"class":926},[907,5485,1198],{"class":913},[907,5487,5488],{"class":926},"bytes, ",[907,5490,2346],{"class":1007},[907,5492,1072],{"class":913},[907,5494,5495],{"class":926},"batch_size)\n",[43,5497,5499],{"id":5498},"prefill-then-decode","Prefill, then decode",[11,5501,5502,5503,5506],{},"The first forward pass reads the 47 prompt tokens at once, at positions 0 to 46. This is the prefill. The causal mask lets all positions be computed in parallel without seeing the future. Each of the 28 layers writes the keys and values of all 47 positions into its KV cache, with a used region of shape ",[868,5504,5505],{},"[1, 8, 47, 128]"," for keys and the same for values. Foundry reserves a larger buffer for the tokens still to come. Only the logits of the last position are used, and they gave \"Paris\".",[11,5508,5509],{},"Every pass after that is a decode step. The runtime feeds only the newest token, and the model runs all 28 layers for that one position. Attention still needs the keys and values of all earlier tokens. It reads them from the cache and appends one new entry per layer. Positions keep counting. \"Paris\" enters at position 47, not 0, and RoPE rotates it for position 47.",[11,5511,5512],{},[850,5513],{"alt":5514,"src":5515},"Prefill at positions 0 to 46, then two decode steps at positions 47 and 48, with the KV cache growing and the measured output","\u002Fassets\u002Fblog\u002Ffrom-safetensors-to-tokens-executing-a-transformer-step-by-step\u002Fprefill-decode.svg",[11,5517,5518,5519,5522,5523,5525],{},"Foundry's generation loop shows both phases. The first ",[868,5520,5521],{},"step"," is the prefill with the whole prompt. Each later ",[868,5524,5521],{}," passes only the last token and the number of positions already seen.",[898,5527,5529],{"className":900,"code":5528,"language":902,"meta":903,"style":903},"let mut tokens = decoder.step(&prompts.concat(), 0)?;\nlet mut past_seen_tokens = prompt_len;\nfor _ in 1..max_new_tokens {\n    if generation.token_ids.iter().all(|row| is_stopped(row, eos_token_id)) {\n        break;\n    }\n    tokens = decoder.step(&tokens, past_seen_tokens)?;\n    generation.token_ids = extend_rows(generation.token_ids, &tokens, eos_token_id);\n    past_seen_tokens = past_seen_tokens.checked_add(1).ok_or_else(|| {\n        error!(message = \"Token position overflows usize.\", past_seen_tokens);\n        GenerateError::PositionOverflow\n    })?;\n}\n",[868,5530,5531,5573,5587,5606,5644,5651,5656,5680,5708,5741,5755,5764,5772],{"__ignoreMap":903},[907,5532,5533,5535,5538,5541,5543,5546,5548,5550,5552,5554,5557,5559,5562,5565,5567,5569,5571],{"class":909,"line":910},[907,5534,940],{"class":913},[907,5536,5537],{"class":913}," mut",[907,5539,5540],{"class":926}," tokens ",[907,5542,946],{"class":913},[907,5544,5545],{"class":926}," decoder",[907,5547,1072],{"class":913},[907,5549,5521],{"class":916},[907,5551,238],{"class":926},[907,5553,1198],{"class":913},[907,5555,5556],{"class":926},"prompts",[907,5558,1072],{"class":913},[907,5560,5561],{"class":916},"concat",[907,5563,5564],{"class":926},"(), ",[907,5566,180],{"class":1007},[907,5568,472],{"class":926},[907,5570,965],{"class":913},[907,5572,927],{"class":926},[907,5574,5575,5577,5579,5582,5584],{"class":909,"line":930},[907,5576,940],{"class":913},[907,5578,5537],{"class":913},[907,5580,5581],{"class":926}," past_seen_tokens ",[907,5583,946],{"class":913},[907,5585,5586],{"class":926}," prompt_len;\n",[907,5588,5589,5592,5595,5598,5601,5603],{"class":909,"line":937},[907,5590,5591],{"class":913},"for",[907,5593,5594],{"class":926}," _ ",[907,5596,5597],{"class":913},"in",[907,5599,5600],{"class":1007}," 1",[907,5602,3667],{"class":913},[907,5604,5605],{"class":926},"max_new_tokens {\n",[907,5607,5608,5610,5613,5615,5618,5620,5622,5624,5626,5629,5631,5633,5636,5638,5641],{"class":909,"line":970},[907,5609,3696],{"class":913},[907,5611,5612],{"class":926}," generation",[907,5614,1072],{"class":913},[907,5616,5617],{"class":926},"token_ids",[907,5619,1072],{"class":913},[907,5621,4510],{"class":916},[907,5623,1149],{"class":926},[907,5625,1072],{"class":913},[907,5627,5628],{"class":916},"all",[907,5630,238],{"class":926},[907,5632,338],{"class":913},[907,5634,5635],{"class":926},"row",[907,5637,338],{"class":913},[907,5639,5640],{"class":916}," is_stopped",[907,5642,5643],{"class":926},"(row, eos_token_id)) {\n",[907,5645,5646,5649],{"class":909,"line":985},[907,5647,5648],{"class":913},"        break",[907,5650,927],{"class":926},[907,5652,5653],{"class":909,"line":1001},[907,5654,5655],{"class":926},"    }\n",[907,5657,5658,5661,5663,5665,5667,5669,5671,5673,5676,5678],{"class":909,"line":1014},[907,5659,5660],{"class":926},"    tokens ",[907,5662,946],{"class":913},[907,5664,5545],{"class":926},[907,5666,1072],{"class":913},[907,5668,5521],{"class":916},[907,5670,238],{"class":926},[907,5672,1198],{"class":913},[907,5674,5675],{"class":926},"tokens, past_seen_tokens)",[907,5677,965],{"class":913},[907,5679,927],{"class":926},[907,5681,5682,5685,5687,5690,5692,5695,5698,5700,5703,5705],{"class":909,"line":1031},[907,5683,5684],{"class":926},"    generation",[907,5686,1072],{"class":913},[907,5688,5689],{"class":926},"token_ids ",[907,5691,946],{"class":913},[907,5693,5694],{"class":916}," extend_rows",[907,5696,5697],{"class":926},"(generation",[907,5699,1072],{"class":913},[907,5701,5702],{"class":926},"token_ids, ",[907,5704,1198],{"class":913},[907,5706,5707],{"class":926},"tokens, eos_token_id);\n",[907,5709,5710,5713,5715,5718,5720,5723,5725,5727,5729,5731,5734,5736,5739],{"class":909,"line":1042},[907,5711,5712],{"class":926},"    past_seen_tokens ",[907,5714,946],{"class":913},[907,5716,5717],{"class":926}," past_seen_tokens",[907,5719,1072],{"class":913},[907,5721,5722],{"class":916},"checked_add",[907,5724,238],{"class":926},[907,5726,318],{"class":1007},[907,5728,472],{"class":926},[907,5730,1072],{"class":913},[907,5732,5733],{"class":916},"ok_or_else",[907,5735,238],{"class":926},[907,5737,5738],{"class":913},"||",[907,5740,3691],{"class":926},[907,5742,5743,5745,5747,5749,5752],{"class":909,"line":1053},[907,5744,5441],{"class":916},[907,5746,5444],{"class":926},[907,5748,946],{"class":913},[907,5750,5751],{"class":959}," \"Token position overflows usize.\"",[907,5753,5754],{"class":926},", past_seen_tokens);\n",[907,5756,5757,5759,5761],{"class":909,"line":1059},[907,5758,5462],{"class":916},[907,5760,920],{"class":913},[907,5762,5763],{"class":916},"PositionOverflow\n",[907,5765,5766,5768,5770],{"class":909,"line":1090},[907,5767,5472],{"class":926},[907,5769,965],{"class":913},[907,5771,927],{"class":926},[907,5773,5775],{"class":909,"line":5774},13,[907,5776,4681],{"class":926},[11,5778,5779,5780,5782],{},"For our question, the model wrote three tokens. \"Paris\" came from the prefill. The first decode step, at position 47, returned \".\" as token 13. The second, at position 48, returned ",[868,5781,878],{},", token 128009, and generation stopped. This token marks the end of the assistant turn.",[11,5784,5785,5786,1072],{},"The next article looks at ",[15,5787,5789],{"href":5788},"\u002Fblog\u002Fthe-hidden-memory-cost-of-llm-inference-kv-cache-explained\u002F","how much memory the KV cache needs",[43,5791,5793],{"id":5792},"the-whole-path-in-one-view","The whole path in one view",[11,5795,5796],{},"The schematic follows the prefill from token IDs to the first answer token. It shows where each matrix multiplication happens and how the tensor shapes change. A reduction combines several numbers into one, such as a sum or a maximum. Pointwise operations work on individual values or pairs, as in the residual adds, SiLU and RoPE.",[11,5798,5799],{},[850,5800],{"alt":5801,"src":5802},"Every operation on the path of one token, from tokenization to the detokenized answer, with its kind and output shape","\u002Fassets\u002Fblog\u002Ffrom-safetensors-to-tokens-executing-a-transformer-step-by-step\u002Ftoken-journey.svg",[11,5804,5805],{},"The residual stream stays 3072 numbers wide between blocks. Inside the MLP, the two projections each produce 8192 values. The LM head produces 128,256 scores, one per vocabulary entry.",[11,5807,5808],{},"We kept the model revision, input IDs and template date fixed for the comparison. The reference run used the stop IDs in the model's generation configuration. All eight module checks passed, and both engines generated the same three token IDs. Their logits differed slightly, so the probabilities in this article come from the Transformers reference run.",[11,5810,5811],{},"The records below include the model revision, software versions, captured scores and comparison results. The module report gives the tolerances and measured differences for each check.",[5813,5814],"json-artifact",{"file":5815,"title":5816},"from-safetensors-to-tokens-executing-a-transformer-step-by-step\u002Fmeasurements.json","Run data",[5813,5818],{"file":5819,"title":5820},"from-safetensors-to-tokens-executing-a-transformer-step-by-step\u002Ffoundry-eval.json","Foundry comparison",[11,5822,5823,5824,1072],{},"For a review of your model-serving architecture, see our ",[15,5825,5827],{"href":5826},"\u002Fservices\u002Fproduction-ai-systems\u002F","AI systems engineering service",[43,5829,5831],{"id":5830},"sources","Sources",[5833,5834,5835,5843,5850,5857,5864,5871,5878],"ul",{},[5836,5837,5838],"li",{},[15,5839,5842],{"href":5840,"rel":5841},"https:\u002F\u002Fhuggingface.co\u002Fmeta-llama\u002FLlama-3.2-3B-Instruct\u002Fblob\u002F0cb88a4f764b7a12671c53f0838cd831a0843b95\u002Fconfig.json",[19],"Llama 3.2 3B Instruct model configuration",[5836,5844,5845],{},[15,5846,5849],{"href":5847,"rel":5848},"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers\u002Fblob\u002Fv5.12.1\u002Fsrc\u002Ftransformers\u002Fmodels\u002Fllama\u002Fmodeling_llama.py",[19],"Llama reference implementation in Transformers",[5836,5851,5852],{},[15,5853,5856],{"href":5854,"rel":5855},"https:\u002F\u002Fgithub.com\u002Fstratorys\u002Ffoundry\u002Ftree\u002Fc147cc3ed22bb97089e0397877ec7a422cfb3f65",[19],"Foundry source used for the comparison",[5836,5858,5859],{},[15,5860,5863],{"href":5861,"rel":5862},"https:\u002F\u002Farxiv.org\u002Fabs\u002F1706.03762",[19],"Attention Is All You Need",[5836,5865,5866],{},[15,5867,5870],{"href":5868,"rel":5869},"https:\u002F\u002Farxiv.org\u002Fabs\u002F2104.09864",[19],"RoFormer and rotary position embeddings",[5836,5872,5873],{},[15,5874,5877],{"href":5875,"rel":5876},"https:\u002F\u002Farxiv.org\u002Fabs\u002F2305.13245",[19],"Grouped-query attention",[5836,5879,5880],{},[15,5881,5884],{"href":5882,"rel":5883},"https:\u002F\u002Fgithub.com\u002Fsafetensors\u002Fsafetensors",[19],"Safetensors source code and format documentation",[65,5886,5887],{},"html pre.shiki code .snl16, html code.shiki .snl16{--shiki-default:#F97583}html pre.shiki code .svObZ, html code.shiki .svObZ{--shiki-default:#B392F0}html pre.shiki code .s95oV, html code.shiki .s95oV{--shiki-default:#E1E4E8}html pre.shiki code .sU2Wk, html code.shiki .sU2Wk{--shiki-default:#9ECBFF}html pre.shiki code .sDLfK, html code.shiki .sDLfK{--shiki-default:#79B8FF}html pre.shiki code .sAwPA, html code.shiki .sAwPA{--shiki-default:#6A737D}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":903,"searchDepth":930,"depth":930,"links":5889},[5890,5891,5892,5893,5894,5895,5896,5897,5898,5899],{"id":45,"depth":930,"text":46},{"id":862,"depth":930,"text":863},{"id":1280,"depth":930,"text":1281},{"id":2447,"depth":930,"text":2448},{"id":3011,"depth":930,"text":3012},{"id":4944,"depth":930,"text":4945},{"id":5177,"depth":930,"text":5178},{"id":5498,"depth":930,"text":5499},{"id":5792,"depth":930,"text":5793},{"id":5830,"depth":930,"text":5831},"\u002Fassets\u002Fblog\u002Ffrom-safetensors-to-tokens-executing-a-transformer-step-by-step\u002Fcover.png","2026-09-21","Follow a prompt through Llama 3.2, from stored weights to attention, softmax and the next token, with measured values and Rust code.","md",false,"concept","en",{},null,"\u002Fblog\u002Ffrom-safetensors-to-tokens-executing-a-transformer-step-by-step","14 min read",{"title":5,"description":5902},{"loc":5909,"images":5913},[5914,5915,5916,5917,5918,5919,5920,5921],{"loc":853},{"loc":1103},{"loc":2078},{"loc":2810},{"loc":4690},{"loc":5344},{"loc":5515},{"loc":5802},"blog\u002Ffrom-safetensors-to-tokens-executing-a-transformer-step-by-step",[5924,5925,5926],"Safetensors","Transformers","LLM inference","a-nsrps9lpQIc6ksXqV5r8iTnrsMBDhiAKyC9WXF33s",[],{"left":5930,"top":5930,"width":5931,"height":5931,"rotate":5930,"vFlip":5904,"hFlip":5904,"body":5932},0,24,"\u003Cpath fill=\"currentColor\" d=\"M12 .297c-6.63 0-12 5.373-12 12c0 5.303 3.438 9.8 8.205 11.385c.6.113.82-.258.82-.577c0-.285-.01-1.04-.015-2.04c-3.338.724-4.042-1.61-4.042-1.61C4.422 18.07 3.633 17.7 3.633 17.7c-1.087-.744.084-.729.084-.729c1.205.084 1.838 1.236 1.838 1.236c1.07 1.835 2.809 1.305 3.495.998c.108-.776.417-1.305.76-1.605c-2.665-.3-5.466-1.332-5.466-5.93c0-1.31.465-2.38 1.235-3.22c-.135-.303-.54-1.523.105-3.176c0 0 1.005-.322 3.3 1.23c.96-.267 1.98-.399 3-.405c1.02.006 2.04.138 3 .405c2.28-1.552 3.285-1.23 3.285-1.23c.645 1.653.24 2.873.12 3.176c.765.84 1.23 1.91 1.23 3.22c0 4.61-2.805 5.625-5.475 5.92c.42.36.81 1.096.81 2.22c0 1.606-.015 2.896-.015 3.286c0 .315.21.69.825.57C20.565 22.092 24 17.592 24 12.297c0-6.627-5.373-12-12-12\"\u002F>",{"left":5930,"top":5930,"width":5931,"height":5931,"rotate":5930,"vFlip":5904,"hFlip":5904,"body":5934,"hidden":933},"\u003Cpath fill=\"currentColor\" d=\"M20.447 20.452h-3.554v-5.569c0-1.328-.027-3.037-1.852-3.037c-1.853 0-2.136 1.445-2.136 2.939v5.667H9.351V9h3.414v1.561h.046c.477-.9 1.637-1.85 3.37-1.85c3.601 0 4.267 2.37 4.267 5.455v6.286zM5.337 7.433a2.06 2.06 0 0 1-2.063-2.065a2.064 2.064 0 1 1 2.063 2.065m1.782 13.019H3.555V9h3.564zM22.225 0H1.771C.792 0 0 .774 0 1.729v20.542C0 23.227.792 24 1.771 24h20.451C23.2 24 24 23.227 24 22.271V1.729C24 .774 23.2 0 22.222 0z\"\u002F>",{"left":5930,"top":5930,"width":5931,"height":5931,"rotate":5930,"vFlip":5904,"hFlip":5904,"body":5936},"\u003Cpath fill=\"currentColor\" d=\"M5.202 2.857C7.954 4.922 10.913 9.11 12 11.358c1.087-2.247 4.046-6.436 6.798-8.501C20.783 1.366 24 .213 24 3.883c0 .732-.42 6.156-.667 7.037c-.856 3.061-3.978 3.842-6.755 3.37c4.854.826 6.089 3.562 3.422 6.299c-5.065 5.196-7.28-1.304-7.847-2.97c-.104-.305-.152-.448-.153-.327c0-.121-.05.022-.153.327c-.568 1.666-2.782 8.166-7.847 2.97c-2.667-2.737-1.432-5.473 3.422-6.3c-2.777.473-5.899-.308-6.755-3.369C.42 10.04 0 4.615 0 3.883c0-3.67 3.217-2.517 5.202-1.026\"\u002F>",{"left":5930,"top":5930,"width":5931,"height":5931,"rotate":5930,"vFlip":5904,"hFlip":5904,"body":5938},"\u003Cpath fill=\"currentColor\" d=\"M14.234 10.162L22.977 0h-2.072l-7.591 8.824L7.251 0H.258l9.168 13.343L.258 24H2.33l8.016-9.318L16.749 24h6.993zm-2.837 3.299l-.929-1.329L3.076 1.56h3.182l5.965 8.532l.929 1.329l7.754 11.09h-3.182z\"\u002F>",1793168446793]