[{"data":1,"prerenderedAt":5004},["ShallowReactive",2],{"blog-\u002Fblog\u002Fthe-hidden-memory-cost-of-llm-inference-kv-cache-explained\u002F":3,"blog-translation-\u002Fblog\u002Fthe-hidden-memory-cost-of-llm-inference-kv-cache-explained\u002F":4977,"blog-series-\u002Fblog\u002Fthe-hidden-memory-cost-of-llm-inference-kv-cache-explained\u002F":4993,"i-simple-icons:github":4994,"i-simple-icons:linkedin":4998,"i-simple-icons:bluesky":5000,"i-simple-icons:x":5002},{"id":4,"title":5,"author":6,"body":7,"cover":4969,"date":4970,"description":4971,"extension":4972,"featured":4973,"kind":4974,"language":4975,"meta":4976,"modified":4977,"navigation":4723,"path":4978,"published":4723,"readingTime":4979,"seo":4980,"seriesKey":4977,"seriesOrder":4977,"seriesTitle":4977,"sitemap":4981,"stem":4987,"tags":4988,"translationKey":4977,"__hash__":4992},"blog\u002Fblog\u002Fthe-hidden-memory-cost-of-llm-inference-kv-cache-explained.md","The hidden memory cost of LLM inference: KV cache explained","Lucas Jahier",{"type":8,"value":9,"toc":4961},"minimark",[10,14,23,28,31,39,42,45,49,52,256,735,972,975,978,988,1420,1681,1968,1979,2573,2576,2583,2586,2590,2593,2596,2605,2608,2611,2617,2620,2624,2881,3393,3396,3399,4023,4026,4035,4041,4048,4051,4461,4464,4516,4519,4525,4528,4537,4540,4543,4845,4852,4879,4882,4886,4889,4898,4901,4910,4914,4917,4920,4923,4926,4946,4955,4958],[11,12,13],"p",{},"Suppose a model loads and answers a short prompt without trouble. Then a long conversation arrives, or several users start generating at the same time, and the server runs out of memory. The weights are the same. More memory is now needed for the work done during those requests.",[11,15,16,17,22],{},"Part of that memory holds the KV cache, data kept from earlier tokens so the model can reuse it. Its size follows directly from how attention works and how the model generates text. Unlike the weights in a ",[18,19,21],"a",{"href":20},"\u002Fblog\u002Finside-llm-model-files-understanding-safetensors-and-gguf\u002F","model file",", this data is built while the model runs.",[24,25,27],"h2",{"id":26},"why-the-next-token-depends-on-earlier-ones","Why the next token depends on earlier ones",[11,29,30],{},"A token is a piece of the model's input, often a word, part of a word or punctuation. The models covered here are Transformer decoders with causal attention. They generate text autoregressively. That means each new token is predicted from the tokens already in the context, including the prompt and the output produced so far.",[11,32,33,34,38],{},"The model first processes the prompt and returns scores for the next possible tokens. The serving engine chooses one, adds it to the context and asks for the next prediction. The same loop continues until generation stops. The ",[18,35,37],{"href":36},"\u002Fblog\u002Ffrom-safetensors-to-tokens-executing-a-transformer-step-by-step\u002F","Transformer walkthrough"," follows a complete run through these steps.",[11,40,41],{},"Every prediction needs information from the existing context. One way to get it is to process that context again on each pass. A cache gives the engine another option. It keeps part of the earlier attention calculations and reuses them when processing the new position.",[11,43,44],{},"This makes the KV cache central to efficient generation in these models. Autoregression still works without it, but the engine repeats calculations for tokens it has already processed. To see which results are worth keeping, we need to look inside attention.",[24,46,48],{"id":47},"what-queries-keys-and-values-do","What queries, keys and values do",[11,50,51],{},"At each layer, a token is represented by a vector, a list of numbers. Attention lets the current position gather information from other positions. It does this through three learned transformations called query, key and value projections. Each projection multiplies the input vector by a matrix, a table of learned numbers stored in the model.",[11,53,54,55,161,162,255],{},"Attention is split into heads. Each head makes its own comparisons and produces its own output vector. For one head at position ",[56,57,69,74,86,90,92],"svg",{"style":58,"className":59,"viewBox":61,"width":62,"height":63,"dataWidth":64,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},"vertical-align: -0.5445454545454546em;",[60],"typst-doc","0 0 4.000 19.000","0.36363636363636365em","1.7272727272727273em","4.000","19.000","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","http:\u002F\u002Fwww.w3.org\u002F1999\u002Fxlink","http:\u002F\u002Fwww.w3.org\u002F1999\u002Fxhtml",[70,71,73],"style",{"type":72},"text\u002Fcss",".typst-text {\n  pointer-events: bounding-box;\n}\n\n.tsel span,\n.tsel {\n  left: 0;\n  position: fixed;\n  text-align: justify;\n  white-space: nowrap;\n  width: 100%;\n  height: 100%;\n  text-align-last: justify;\n  color: transparent;\n  white-space: pre;\n}\n.tsel span::-moz-selection,\n.tsel::-moz-selection {\n  color: transparent;\n  background: #7db9dea0;\n}\n.tsel span::selection,\n.tsel::selection {\n  color: transparent;\n  background: #7db9dea0;\n}\n.pseudo-link {\n  fill: transparent;\n  cursor: pointer;\n  pointer-events: all;\n}\nsvg {\n  fill: none;\n}\n\n.outline_glyph path,\npath.outline_glyph {\n  fill: var(--glyph_fill);\n  stroke: var(--glyph_stroke);\n}\n\n.outline_glyph path,\npath.outline_glyph {\n  transition: 0.2s fill stroke;\n}\n.hover .typst-text {\n  --glyph_fill: #66bab7;\n  --glyph_stroke: #66bab7;\n}\n\n.typst-jump-ripple,\n.typst-debug-react-ripple {\n  width: 0;\n  height: 0;\n  background-color: transparent;\n  position: absolute;\n  border-radius: 50%;\n}\n.typst-jump-ripple {\n  border: 1px solid #66bab7;\n}\n.typst-debug-react-ripple {\n  border: 1px solid #cb1b45;\n}\n@keyframes typst-jump-ripple-effect {\n  to {\n    width: 10vw;\n    height: 10vw;\n    opacity: 0.01;\n    margin: -5vw;\n  }\n}\n@keyframes typst-debug-react-ripple-effect {\n  to {\n    width: 3vw;\n    height: 3vw;\n    opacity: 0.01;\n    margin: -1.5vw;\n  }\n}\n",[75,76,79],"defs",{"className":77},[78],"glyph",[80,81],"path",{"id":82,"className":83,"d":85},"gl3Hi1RoL",[84],"outline_glyph","M 330 419 C 330 428 320 433 299 433 L 218 433 C 244 537 257 591 257 595 C 257 616 246 626 225 626 C 202 626 188 613 182 587 L 145 433 L 56 433 C 34 433 23 432 23 411 C 23 400 33 395 54 395 L 135 395 C 86 200 62 97 62 84 C 62 28 100 -11 156 -11 C 194 -11 227 6 256 41 C 280 70 297 98 308 125 C 312 136 314 142 314 145 C 314 154 309 159 299 159 C 291 159 285 154 281 143 C 247 60 206 19 157 19 C 140 19 131 33 131 60 C 131 75 133 91 137 107 L 208 395 L 297 395 C 323 395 330 397 330 419 Z ",[75,87],{"className":88},[89],"clip-path",[70,91],{"type":72},[93,94,101],"g",{"className":95,"transform":97,"dataTid":98,"dataPageWidth":99,"dataPageHeight":100},[96],"typst-page","translate(0, 0)","pKXVZzGGdbwEsb+MU5PmHu8","4","19",[93,102,106],{"dataTid":103,"className":104},"gKXVZzGGdbwEsb+MU5PmHu8",[105],"typst-group",[93,107,110],{"transform":108,"dataTid":109},"translate(0.000,0.000)","pRoh4UK5Oo646CzJEGnqHa4",[93,111,114,153],{"dataTid":112,"className":113},"gRoh4UK5Oo646CzJEGnqHa4",[105],[93,115,117],{"transform":108,"dataTid":116},"pwcSiCSxgnQQitTfipb3dOw",[93,118,121],{"dataTid":119,"className":120},"gwcSiCSxgnQQitTfipb3dOw",[105],[93,122,125],{"transform":123,"dataTid":124},"translate(0.000,13.013)","pmE4OJGGgR4kBPmeTKczjCU",[93,126,132,137],{"dataTid":127,"className":128,"fill":130,"transform":131},"gmE4OJGGgR4kBPmeTKczjCU",[129],"typst-text","#000","scale(0.011,-0.011)",[133,134],"use",{"x":135,"y":135,"href":136},"0","#gl3Hi1RoL",[93,138,140],{"transform":139},"scale(16,-16)",[141,142,146],"foreignObject",{"x":135,"y":143,"width":144,"height":145},"-50.38","22.56","62.50",[147,148,152],"h5:div",{"className":149,"style":151},[150],"tsel","font-size: 62px","𝑡",[93,154,156],{"transform":108,"dataTid":155},"p+nF9nphQ3U0gANvmKrhITw",[93,157],{"dataTid":158,"className":159,"dataHint":18},"g+nF9nphQ3U0gANvmKrhITw",[160],"typst-content-hint",", let ",[56,163,168,170,183,186,188],{"style":58,"className":164,"viewBox":165,"width":166,"height":63,"dataWidth":167,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],"0 0 11.000 19.000","1em","11.000",[70,169,73],{"type":72},[75,171,173,178],{"className":172},[78],[80,174],{"id":175,"className":176,"d":177},"gl3Hi1RED",[84],"M 512 138 C 489 59 457 19 415 19 C 402 19 395 28 395 47 C 395 62 401 85 413 116 C 453 223 473 296 473 335 C 473 406 427 445 356 445 C 303 445 257 423 218 380 L 291 679 C 289 688 287 694 274 694 C 241 694 168 685 155 684 C 140 682 132 675 132 660 C 132 650 141 645 159 645 C 180 645 203 646 205 632 L 59 43 C 56 32 55 25 55 21 C 55 0 66 -11 87 -11 C 107 -11 121 -1 128 18 L 168 182 C 180 231 189 268 196 293 C 199 300 207 314 220 334 C 256 388 300 415 353 415 C 386 415 402 393 402 350 C 402 309 382 236 342 130 C 333 107 329 89 329 74 C 329 26 365 -11 413 -11 C 457 -11 492 14 517 65 C 536 105 546 132 546 145 C 546 154 541 159 530 159 C 527 159 512 150 512 138 Z ",[80,179],{"id":180,"className":181,"d":182},"gl3Hi1d8T",[84],"M 380 416 C 380 427 369 433 347 433 L 259 433 L 294 575 C 294 577 297 587 297 589 C 297 613 285 625 260 625 C 230 625 213 600 206 566 L 174 433 L 77 433 C 54 433 42 430 42 408 C 42 396 53 390 75 390 L 163 390 L 107 162 C 97 122 92 97 92 88 C 92 29 141 -10 200 -10 C 251 -10 295 14 330 63 C 359 104 374 131 374 145 C 374 156 368 162 355 162 C 350 162 346 161 343 160 C 337 153 333 146 331 141 C 309 92 263 26 203 26 C 183 26 173 41 173 70 C 173 81 175 94 179 110 L 248 390 L 345 390 C 368 390 380 395 380 416 Z ",[75,184],{"className":185},[89],[70,187],{"type":72},[93,189,193],{"className":190,"transform":97,"dataTid":191,"dataPageWidth":192,"dataPageHeight":100},[96],"pPo\u002FlEG8MMNwx5G48K30Xo0","11",[93,194,197],{"dataTid":195,"className":196},"gPo\u002FlEG8MMNwx5G48K30Xo0",[105],[93,198,200],{"transform":108,"dataTid":199},"pwmyCRX77LmUETLVrK9QEzk",[93,201,204,250],{"dataTid":202,"className":203},"gwmyCRX77LmUETLVrK9QEzk",[105],[93,205,207],{"transform":108,"dataTid":206},"pc4DFbvmsAkwWPRFn9brXLc",[93,208,211,230],{"dataTid":209,"className":210},"gc4DFbvmsAkwWPRFn9brXLc",[105],[93,212,214],{"transform":123,"dataTid":213},"pAtC6VGx45DAmDDu5SrFZW0",[93,215,218,221],{"dataTid":216,"className":217,"fill":130,"transform":131},"gAtC6VGx45DAmDDu5SrFZW0",[129],[133,219],{"x":135,"y":135,"href":220},"#gl3Hi1RED",[93,222,223],{"transform":139},[141,224,226],{"x":135,"y":143,"width":225,"height":145},"36.00",[147,227,229],{"className":228,"style":151},[150],"ℎ",[93,231,234],{"transform":232,"dataTid":233},"translate(6.336,15.730)","pvpm58+yG7Q4vnT2JFfVj+w",[93,235,239,242],{"dataTid":236,"className":237,"fill":130,"transform":238},"gvpm58+yG7Q4vnT2JFfVj+w",[129],"scale(0.0077,-0.0077)",[133,240],{"x":135,"y":135,"href":241},"#gl3Hi1d8T",[93,243,244],{"transform":139},[141,245,247],{"x":135,"y":143,"width":246,"height":145},"27.00",[147,248,152],{"className":249,"style":151},[150],[93,251,252],{"transform":108,"dataTid":155},[93,253],{"dataTid":158,"className":254,"dataHint":18},[160]," be the input vector. The three projections give",[56,257,265,267,316,319,321],{"style":258,"className":259,"viewBox":260,"width":261,"height":262,"dataWidth":263,"dataHeight":264,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},"vertical-align: -NaNem;; display: block; margin: 0 auto;",[60],"0 0 53.000 48.000","4.818181818181818em","4.363636363636363em","53.000","48.000",[70,266,73],{"type":72},[75,268,270,275,278,283,288,293,298,303,308,313],{"className":269},[78],[80,271],{"id":272,"className":273,"d":274},"gl3Hi1R4",[84],"M 698 367 L 80 367 C 64 367 56 359 56 344 C 56 329 64 321 80 321 L 698 321 C 714 321 722 329 722 344 C 722 356 711 367 698 367 Z M 698 179 L 80 179 C 64 179 56 171 56 156 C 56 141 64 133 80 133 L 698 133 C 714 133 722 141 722 156 C 722 169 711 179 698 179 Z ",[80,276],{"id":175,"className":277,"d":177},[84],[80,279],{"id":280,"className":281,"d":282},"gl3Hi1QQL",[84],"M 956 680 C 937 680 872 683 853 683 C 838 683 830 675 830 660 C 830 650 836 645 848 644 C 885 643 903 632 903 610 C 903 604 899 594 891 579 L 629 125 L 594 602 C 590 638 614 644 659 644 C 680 644 690 652 690 668 C 690 678 684 683 671 683 C 649 683 572 680 550 680 C 531 680 465 683 446 683 C 431 683 423 675 423 659 C 423 649 432 644 451 644 C 482 644 498 637 499 623 L 505 548 L 261 125 L 225 613 C 222 638 264 644 296 644 C 313 644 321 652 321 668 C 321 678 315 683 303 683 C 281 683 203 680 181 680 C 162 680 96 683 77 683 C 62 683 55 675 55 660 C 55 649 65 644 84 644 C 126 644 130 642 132 612 L 177 7 C 178 -12 185 -22 197 -22 C 210 -20 215 -17 223 -2 C 224 -1 224 0 224 1 L 509 494 L 545 7 C 546 -12 553 -22 565 -22 C 575 -22 584 -15 592 -1 L 918 562 C 945 611 964 639 1025 644 C 1043 646 1048 651 1048 668 C 1048 678 1043 683 1032 683 C 1018 683 970 680 956 680 Z ",[80,284],{"id":285,"className":286,"d":287},"gl3Hi1REL",[84],"M 409 353 C 409 327 423 314 450 314 C 485 314 508 345 508 379 C 508 418 476 445 437 445 C 392 445 344 415 291 356 C 250 311 217 282 190 269 L 291 679 C 289 688 287 694 274 694 C 242 694 166 685 154 684 C 139 682 132 675 132 660 C 132 650 141 645 159 645 C 178 645 204 646 204 632 L 59 43 C 56 32 55 25 55 21 C 55 0 66 -11 87 -11 C 104 -11 117 -3 124 12 C 129 21 147 92 179 226 C 231 221 286 196 286 146 C 286 131 279 101 279 91 C 279 34 316 -11 373 -11 C 431 -11 470 41 490 145 C 490 154 485 159 475 159 C 466 159 460 152 457 138 C 435 59 408 19 375 19 C 357 19 348 33 348 61 C 348 77 360 131 360 147 C 360 204 314 239 221 253 C 244 269 270 292 298 322 C 326 352 346 371 359 382 C 386 404 412 415 435 415 C 445 415 453 413 459 409 C 432 404 409 379 409 353 Z ",[80,289],{"id":290,"className":291,"d":292},"gl3Hi1RcL",[84],"M 372 377 C 352 420 321 442 280 442 C 215 442 158 409 109 342 C 63 280 40 216 40 149 C 40 62 90 -13 173 -13 C 209 -13 245 4 282 38 L 241 -126 C 238 -141 229 -150 215 -153 C 210 -154 197 -155 174 -155 C 168 -156 164 -156 162 -156 C 153 -157 148 -165 148 -179 L 148 -183 C 151 -190 157 -194 165 -194 C 184 -194 244 -191 263 -191 C 282 -191 345 -194 364 -194 C 378 -194 385 -186 385 -170 C 385 -160 375 -155 356 -155 C 338 -155 313 -156 313 -144 C 313 -140 314 -133 317 -123 L 452 427 C 452 436 447 441 438 441 C 420 441 382 391 372 377 Z M 340 373 C 349 352 354 338 354 329 C 354 328 353 322 351 313 L 327 216 C 308 144 299 107 298 105 C 279 70 224 16 176 16 C 137 16 117 46 117 105 C 117 151 148 263 162 300 C 181 347 225 412 280 412 C 307 412 327 399 340 373 Z ",[80,294],{"id":295,"className":296,"d":297},"gl3Hi1RwL",[84],"M 369 391 C 369 383 378 370 394 350 C 410 330 418 307 418 281 C 418 252 404 203 375 134 C 354 86 306 18 247 18 C 201 18 178 45 178 100 C 178 139 197 208 235 307 C 243 328 247 345 247 357 C 247 407 213 442 163 442 C 119 442 84 417 59 367 C 39 326 29 300 29 287 C 29 278 34 273 45 273 C 58 273 60 279 64 293 C 87 373 119 413 160 413 C 173 413 180 404 180 385 C 180 369 175 347 164 318 C 127 219 108 152 108 115 C 108 64 125 29 159 10 C 186 -4 214 -11 243 -11 C 332 -11 394 85 420 162 C 452 256 468 325 468 369 C 468 418 452 442 421 442 C 396 442 369 416 369 391 Z ",[80,299],{"id":300,"className":301,"d":302},"gl3Hi1bwT",[84],"M 894 679 C 872 679 796 683 774 683 C 757 683 749 674 749 657 C 752 647 753 641 768 640 C 782 639 789 635 789 630 C 789 619 775 604 747 584 L 342 300 L 418 601 C 423 621 433 633 448 638 C 453 639 470 640 501 640 C 524 640 535 645 535 664 C 535 677 528 683 515 683 C 490 683 410 679 385 679 C 338 679 276 683 254 683 C 238 683 230 674 230 657 C 230 646 242 640 266 640 C 301 640 322 642 322 628 L 186 82 C 181 62 171 50 156 46 C 151 44 133 43 103 43 C 80 43 69 40 69 17 C 72 6 73 0 89 0 C 113 0 194 4 218 4 C 265 4 328 0 350 0 C 366 0 374 9 374 26 C 374 35 371 40 364 41 C 353 42 345 43 339 43 C 309 44 282 42 282 56 L 328 244 L 488 356 L 630 82 C 634 74 636 68 636 65 C 636 50 622 43 594 43 C 573 43 565 36 565 17 C 568 6 569 0 585 0 C 609 0 688 4 712 4 C 731 4 790 0 809 0 C 825 0 833 8 833 25 C 833 37 824 43 807 43 C 776 43 755 53 745 74 C 687 187 628 298 571 412 L 592 429 L 692 499 C 766 550 807 579 815 584 C 870 618 916 637 953 640 C 970 641 979 650 979 665 C 979 676 973 682 962 682 C 950 682 906 679 894 679 Z ",[80,304],{"id":305,"className":306,"d":307},"gl3Hi1cIT",[84],"M 361 -22 C 392 -22 426 -18 461 -9 L 461 -57 C 461 -94 467 -124 478 -149 C 491 -179 518 -194 559 -194 C 621 -194 671 -147 696 -103 C 725 -53 740 -18 740 2 C 740 14 734 20 722 20 C 713 20 707 15 705 5 C 690 -41 645 -82 586 -82 C 535 -82 523 -44 512 8 C 599 39 673 93 734 169 C 797 249 829 334 829 423 C 829 507 801 575 746 626 C 691 677 622 703 538 703 C 460 703 385 681 312 638 C 193 567 69 426 69 254 C 69 88 192 -22 361 -22 Z M 725 450 C 725 391 710 324 679 249 C 644 160 585 95 502 54 C 483 122 449 156 400 156 C 344 156 286 109 286 53 C 286 46 286 39 287 34 C 211 64 173 129 173 229 C 173 342 210 444 283 534 C 335 598 426 663 530 663 C 652 663 725 575 725 450 Z M 400 120 C 438 120 458 92 461 35 C 430 24 399 18 368 18 C 337 18 321 30 321 53 C 321 91 362 120 400 120 Z ",[80,309],{"id":310,"className":311,"d":312},"gl3Hi1ccT",[84],"M 764 679 C 745 679 680 683 661 683 C 645 683 637 674 637 657 C 637 646 644 640 659 639 C 684 638 696 630 696 615 C 696 607 690 594 679 577 L 359 107 L 271 603 C 270 610 269 614 269 617 C 269 632 288 639 327 639 C 348 639 361 644 361 665 L 361 669 C 359 678 352 683 340 683 C 317 683 236 679 213 679 C 192 679 124 683 103 683 C 87 683 79 674 79 657 C 79 645 89 639 108 639 C 151 639 160 642 165 611 L 273 10 C 276 -10 286 -20 301 -20 C 315 -20 327 -12 338 3 L 716 559 C 750 609 790 636 835 640 C 853 641 861 648 861 665 C 861 677 855 683 843 683 C 829 683 778 679 764 679 Z ",[80,314],{"id":180,"className":315,"d":182},[84],[75,317],{"className":318},[89],[70,320],{"type":72},[93,322,327],{"className":323,"transform":97,"dataTid":324,"dataPageWidth":325,"dataPageHeight":326},[96],"pBVdzTbAVDHkkSGJ5bwKjkg","53","48",[93,328,331],{"dataTid":329,"className":330},"gBVdzTbAVDHkkSGJ5bwKjkg",[105],[93,332,334],{"transform":108,"dataTid":333},"pBYZIPx\u002F10lgdSaLzLt66lI",[93,335,338,476,603,730],{"dataTid":336,"className":337},"gBYZIPx\u002F10lgdSaLzLt66lI",[105],[93,339,342],{"transform":340,"dataTid":341},"translate(0.824,0.000)","pciAUcAB+fFY81jQ+zCe8e8",[93,343,346,366,381,401,416,431,451,471],{"dataTid":344,"className":345},"gciAUcAB+fFY81jQ+zCe8e8",[105],[93,347,350],{"transform":348,"dataTid":349},"translate(0.000,7.634)","pYQO\u002F9+Pe7WQXmIuvd3NKGE",[93,351,354,357],{"dataTid":352,"className":353,"fill":130,"transform":131},"gYQO\u002F9+Pe7WQXmIuvd3NKGE",[129],[133,355],{"x":135,"y":135,"href":356},"#gl3Hi1RcL",[93,358,359],{"transform":139},[141,360,362],{"x":135,"y":143,"width":361,"height":145},"30.00",[147,363,365],{"className":364,"style":151},[150],"𝑞",[93,367,369],{"transform":368,"dataTid":233},"translate(4.906,10.351)",[93,370,372,374],{"dataTid":236,"className":371,"fill":130,"transform":238},[129],[133,373],{"x":135,"y":135,"href":241},[93,375,376],{"transform":139},[141,377,378],{"x":135,"y":143,"width":246,"height":145},[147,379,152],{"className":380,"style":151},[150],[93,382,385],{"transform":383,"dataTid":384},"translate(11.904,7.634)","pTVgzH5qlGVoRBbtwYAF8r4",[93,386,389,392],{"dataTid":387,"className":388,"fill":130,"transform":131},"gTVgzH5qlGVoRBbtwYAF8r4",[129],[133,390],{"x":135,"y":135,"href":391},"#gl3Hi1R4",[93,393,394],{"transform":139},[141,395,397],{"x":135,"y":143,"width":396,"height":145},"48.62",[147,398,400],{"className":399,"style":151},[150],"=",[93,402,404],{"transform":403,"dataTid":213},"translate(23.518,7.634)",[93,405,407,409],{"dataTid":216,"className":406,"fill":130,"transform":131},[129],[133,408],{"x":135,"y":135,"href":220},[93,410,411],{"transform":139},[141,412,413],{"x":135,"y":143,"width":225,"height":145},[147,414,229],{"className":415,"style":151},[150],[93,417,419],{"transform":418,"dataTid":233},"translate(29.854,10.351)",[93,420,422,424],{"dataTid":236,"className":421,"fill":130,"transform":238},[129],[133,423],{"x":135,"y":135,"href":241},[93,425,426],{"transform":139},[141,427,428],{"x":135,"y":143,"width":246,"height":145},[147,429,152],{"className":430,"style":151},[150],[93,432,435],{"transform":433,"dataTid":434},"translate(33.796,7.634)","p1J330TwXsYwqOESbbfpOj0",[93,436,439,442],{"dataTid":437,"className":438,"fill":130,"transform":131},"g1J330TwXsYwqOESbbfpOj0",[129],[133,440],{"x":135,"y":135,"href":441},"#gl3Hi1QQL",[93,443,444],{"transform":139},[141,445,447],{"x":135,"y":143,"width":446,"height":145},"67.25",[147,448,450],{"className":449,"style":151},[150],"𝑊",[93,452,455],{"transform":453,"dataTid":454},"translate(43.080,10.351)","ptpz26kUxT2Ei4\u002F7Km0K+O4",[93,456,459,462],{"dataTid":457,"className":458,"fill":130,"transform":238},"gtpz26kUxT2Ei4\u002F7Km0K+O4",[129],[133,460],{"x":135,"y":135,"href":461},"#gl3Hi1cIT",[93,463,464],{"transform":139},[141,465,467],{"x":135,"y":143,"width":466,"height":145},"56.25",[147,468,470],{"className":469,"style":151},[150],"𝑄",[93,472,473],{"transform":108,"dataTid":155},[93,474],{"dataTid":158,"className":475,"dataHint":18},[160],[93,477,480],{"transform":478,"dataTid":479},"translate(0.000,18.995)","pZUvUK3CL+Wsqed2+taqoKA",[93,481,484,503,518,533,548,563,578,598],{"dataTid":482,"className":483},"gZUvUK3CL+Wsqed2+taqoKA",[105],[93,485,487],{"transform":348,"dataTid":486},"pzmFErrJbosg3xUXBw61U8E",[93,488,491,494],{"dataTid":489,"className":490,"fill":130,"transform":131},"gzmFErrJbosg3xUXBw61U8E",[129],[133,492],{"x":135,"y":135,"href":493},"#gl3Hi1REL",[93,495,496],{"transform":139},[141,497,499],{"x":135,"y":143,"width":498,"height":145},"33.50",[147,500,502],{"className":501,"style":151},[150],"𝑘",[93,504,506],{"transform":505,"dataTid":233},"translate(5.731,10.351)",[93,507,509,511],{"dataTid":236,"className":508,"fill":130,"transform":238},[129],[133,510],{"x":135,"y":135,"href":241},[93,512,513],{"transform":139},[141,514,515],{"x":135,"y":143,"width":246,"height":145},[147,516,152],{"className":517,"style":151},[150],[93,519,521],{"transform":520,"dataTid":384},"translate(12.729,7.634)",[93,522,524,526],{"dataTid":387,"className":523,"fill":130,"transform":131},[129],[133,525],{"x":135,"y":135,"href":391},[93,527,528],{"transform":139},[141,529,530],{"x":135,"y":143,"width":396,"height":145},[147,531,400],{"className":532,"style":151},[150],[93,534,536],{"transform":535,"dataTid":213},"translate(24.343,7.634)",[93,537,539,541],{"dataTid":216,"className":538,"fill":130,"transform":131},[129],[133,540],{"x":135,"y":135,"href":220},[93,542,543],{"transform":139},[141,544,545],{"x":135,"y":143,"width":225,"height":145},[147,546,229],{"className":547,"style":151},[150],[93,549,551],{"transform":550,"dataTid":233},"translate(30.679,10.351)",[93,552,554,556],{"dataTid":236,"className":553,"fill":130,"transform":238},[129],[133,555],{"x":135,"y":135,"href":241},[93,557,558],{"transform":139},[141,559,560],{"x":135,"y":143,"width":246,"height":145},[147,561,152],{"className":562,"style":151},[150],[93,564,566],{"transform":565,"dataTid":434},"translate(34.621,7.634)",[93,567,569,571],{"dataTid":437,"className":568,"fill":130,"transform":131},[129],[133,570],{"x":135,"y":135,"href":441},[93,572,573],{"transform":139},[141,574,575],{"x":135,"y":143,"width":446,"height":145},[147,576,450],{"className":577,"style":151},[150],[93,579,582],{"transform":580,"dataTid":581},"translate(43.905,10.351)","pe0jbh2OazrQSo4iGkvaRSY",[93,583,586,589],{"dataTid":584,"className":585,"fill":130,"transform":238},"ge0jbh2OazrQSo4iGkvaRSY",[129],[133,587],{"x":135,"y":135,"href":588},"#gl3Hi1bwT",[93,590,591],{"transform":139},[141,592,594],{"x":135,"y":143,"width":593,"height":145},"62.94",[147,595,597],{"className":596,"style":151},[150],"𝐾",[93,599,600],{"transform":108,"dataTid":155},[93,601],{"dataTid":158,"className":602,"dataHint":18},[160],[93,604,607],{"transform":605,"dataTid":606},"translate(0.652,36.573)","pmrV+dfIUQ3Iz+V4Fea\u002FU14",[93,608,611,630,645,660,675,690,705,725],{"dataTid":609,"className":610},"gmrV+dfIUQ3Iz+V4Fea\u002FU14",[105],[93,612,614],{"transform":348,"dataTid":613},"p3PeuD\u002FFdtjkiEBkBGm4gw8",[93,615,618,621],{"dataTid":616,"className":617,"fill":130,"transform":131},"g3PeuD\u002FFdtjkiEBkBGm4gw8",[129],[133,619],{"x":135,"y":135,"href":620},"#gl3Hi1RwL",[93,622,623],{"transform":139},[141,624,626],{"x":135,"y":143,"width":625,"height":145},"31.00",[147,627,629],{"className":628,"style":151},[150],"𝑣",[93,631,633],{"transform":632,"dataTid":233},"translate(5.335,10.351)",[93,634,636,638],{"dataTid":236,"className":635,"fill":130,"transform":238},[129],[133,637],{"x":135,"y":135,"href":241},[93,639,640],{"transform":139},[141,641,642],{"x":135,"y":143,"width":246,"height":145},[147,643,152],{"className":644,"style":151},[150],[93,646,648],{"transform":647,"dataTid":384},"translate(12.333,7.634)",[93,649,651,653],{"dataTid":387,"className":650,"fill":130,"transform":131},[129],[133,652],{"x":135,"y":135,"href":391},[93,654,655],{"transform":139},[141,656,657],{"x":135,"y":143,"width":396,"height":145},[147,658,400],{"className":659,"style":151},[150],[93,661,663],{"transform":662,"dataTid":213},"translate(23.947,7.634)",[93,664,666,668],{"dataTid":216,"className":665,"fill":130,"transform":131},[129],[133,667],{"x":135,"y":135,"href":220},[93,669,670],{"transform":139},[141,671,672],{"x":135,"y":143,"width":225,"height":145},[147,673,229],{"className":674,"style":151},[150],[93,676,678],{"transform":677,"dataTid":233},"translate(30.283,10.351)",[93,679,681,683],{"dataTid":236,"className":680,"fill":130,"transform":238},[129],[133,682],{"x":135,"y":135,"href":241},[93,684,685],{"transform":139},[141,686,687],{"x":135,"y":143,"width":246,"height":145},[147,688,152],{"className":689,"style":151},[150],[93,691,693],{"transform":692,"dataTid":434},"translate(34.225,7.634)",[93,694,696,698],{"dataTid":437,"className":695,"fill":130,"transform":131},[129],[133,697],{"x":135,"y":135,"href":441},[93,699,700],{"transform":139},[141,701,702],{"x":135,"y":143,"width":446,"height":145},[147,703,450],{"className":704,"style":151},[150],[93,706,709],{"transform":707,"dataTid":708},"translate(43.509,10.351)","pwONjGRyhSBQE6HwQZu3e88",[93,710,713,716],{"dataTid":711,"className":712,"fill":130,"transform":238},"gwONjGRyhSBQE6HwQZu3e88",[129],[133,714],{"x":135,"y":135,"href":715},"#gl3Hi1ccT",[93,717,718],{"transform":139},[141,719,721],{"x":135,"y":143,"width":720,"height":145},"55.56",[147,722,724],{"className":723,"style":151},[150],"𝑉",[93,726,727],{"transform":108,"dataTid":155},[93,728],{"dataTid":158,"className":729,"dataHint":18},[160],[93,731,732],{"transform":108,"dataTid":155},[93,733],{"dataTid":158,"className":734,"dataHint":18},[160],[11,736,737,738,817,818,896,897,971],{},"The matrices ",[56,739,744,746,755,758,760],{"style":58,"className":740,"viewBox":741,"width":742,"height":63,"dataWidth":743,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],"0 0 17.000 19.000","1.5454545454545454em","17.000",[70,745,73],{"type":72},[75,747,749,752],{"className":748},[78],[80,750],{"id":280,"className":751,"d":282},[84],[80,753],{"id":305,"className":754,"d":307},[84],[75,756],{"className":757},[89],[70,759],{"type":72},[93,761,765],{"className":762,"transform":97,"dataTid":763,"dataPageWidth":764,"dataPageHeight":100},[96],"pMS2qnzQP46M1fM\u002Fm2\u002FL1FM","17",[93,766,769],{"dataTid":767,"className":768},"gMS2qnzQP46M1fM\u002Fm2\u002FL1FM",[105],[93,770,772],{"transform":108,"dataTid":771},"pYylvaUhX9Xgnx8DexnPfmw",[93,773,776,812],{"dataTid":774,"className":775},"gYylvaUhX9Xgnx8DexnPfmw",[105],[93,777,779],{"transform":108,"dataTid":778},"pAkL1Fdkc0HgOWZc2OK12ME",[93,780,783,797],{"dataTid":781,"className":782},"gAkL1Fdkc0HgOWZc2OK12ME",[105],[93,784,785],{"transform":123,"dataTid":434},[93,786,788,790],{"dataTid":437,"className":787,"fill":130,"transform":131},[129],[133,789],{"x":135,"y":135,"href":441},[93,791,792],{"transform":139},[141,793,794],{"x":135,"y":143,"width":446,"height":145},[147,795,450],{"className":796,"style":151},[150],[93,798,800],{"transform":799,"dataTid":454},"translate(9.284,15.730)",[93,801,803,805],{"dataTid":457,"className":802,"fill":130,"transform":238},[129],[133,804],{"x":135,"y":135,"href":461},[93,806,807],{"transform":139},[141,808,809],{"x":135,"y":143,"width":466,"height":145},[147,810,470],{"className":811,"style":151},[150],[93,813,814],{"transform":108,"dataTid":155},[93,815],{"dataTid":158,"className":816,"dataHint":18},[160],", ",[56,819,824,826,835,838,840],{"style":58,"className":820,"viewBox":821,"width":822,"height":63,"dataWidth":823,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],"0 0 18.000 19.000","1.6363636363636365em","18.000",[70,825,73],{"type":72},[75,827,829,832],{"className":828},[78],[80,830],{"id":280,"className":831,"d":282},[84],[80,833],{"id":300,"className":834,"d":302},[84],[75,836],{"className":837},[89],[70,839],{"type":72},[93,841,845],{"className":842,"transform":97,"dataTid":843,"dataPageWidth":844,"dataPageHeight":100},[96],"pm1RY14gjrfAyceXwiV7qqU","18",[93,846,849],{"dataTid":847,"className":848},"gm1RY14gjrfAyceXwiV7qqU",[105],[93,850,852],{"transform":108,"dataTid":851},"pqZhNWlq1FnEFEFtWtLruFo",[93,853,856,891],{"dataTid":854,"className":855},"gqZhNWlq1FnEFEFtWtLruFo",[105],[93,857,859],{"transform":108,"dataTid":858},"pDdoZ8J13Gk82m1G2U0d0hY",[93,860,863,877],{"dataTid":861,"className":862},"gDdoZ8J13Gk82m1G2U0d0hY",[105],[93,864,865],{"transform":123,"dataTid":434},[93,866,868,870],{"dataTid":437,"className":867,"fill":130,"transform":131},[129],[133,869],{"x":135,"y":135,"href":441},[93,871,872],{"transform":139},[141,873,874],{"x":135,"y":143,"width":446,"height":145},[147,875,450],{"className":876,"style":151},[150],[93,878,879],{"transform":799,"dataTid":581},[93,880,882,884],{"dataTid":584,"className":881,"fill":130,"transform":238},[129],[133,883],{"x":135,"y":135,"href":588},[93,885,886],{"transform":139},[141,887,888],{"x":135,"y":143,"width":593,"height":145},[147,889,597],{"className":890,"style":151},[150],[93,892,893],{"transform":108,"dataTid":155},[93,894],{"dataTid":158,"className":895,"dataHint":18},[160]," and ",[56,898,900,902,911,914,916],{"style":58,"className":899,"viewBox":741,"width":742,"height":63,"dataWidth":743,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],[70,901,73],{"type":72},[75,903,905,908],{"className":904},[78],[80,906],{"id":280,"className":907,"d":282},[84],[80,909],{"id":310,"className":910,"d":312},[84],[75,912],{"className":913},[89],[70,915],{"type":72},[93,917,920],{"className":918,"transform":97,"dataTid":919,"dataPageWidth":764,"dataPageHeight":100},[96],"p0\u002FhDad\u002FkhiMjcp4McHbgTc",[93,921,924],{"dataTid":922,"className":923},"g0\u002FhDad\u002FkhiMjcp4McHbgTc",[105],[93,925,927],{"transform":108,"dataTid":926},"pjiM9hkqqSsMxVl3yHmAtUQ",[93,928,931,966],{"dataTid":929,"className":930},"gjiM9hkqqSsMxVl3yHmAtUQ",[105],[93,932,934],{"transform":108,"dataTid":933},"ppCNdMGVPjOIuTp77jwhUR8",[93,935,938,952],{"dataTid":936,"className":937},"gpCNdMGVPjOIuTp77jwhUR8",[105],[93,939,940],{"transform":123,"dataTid":434},[93,941,943,945],{"dataTid":437,"className":942,"fill":130,"transform":131},[129],[133,944],{"x":135,"y":135,"href":441},[93,946,947],{"transform":139},[141,948,949],{"x":135,"y":143,"width":446,"height":145},[147,950,450],{"className":951,"style":151},[150],[93,953,954],{"transform":799,"dataTid":708},[93,955,957,959],{"dataTid":711,"className":956,"fill":130,"transform":238},[129],[133,958],{"x":135,"y":135,"href":715},[93,960,961],{"transform":139},[141,962,963],{"x":135,"y":143,"width":720,"height":145},[147,964,724],{"className":965,"style":151},[150],[93,967,968],{"transform":108,"dataTid":155},[93,969],{"dataTid":158,"className":970,"dataHint":18},[160]," stay fixed during inference. The resulting vectors depend on the tokens being processed. The lowercase letters above describe one position. Uppercase Q, K and V usually refer to collections of those vectors. Some models also transform queries and keys to include position information, for example with RoPE. The equations above show only the projections.",[11,973,974],{},"The query describes what the current position will compare against. Each key provides the numbers used in that comparison. Each value provides information that can be passed into the result. All positions produce keys and values, including the current one.",[11,976,977],{},"The head compares the current query with each available key using a dot product. It multiplies the numbers in matching positions in the two vectors, then adds the results. This gives one score per position. A larger score gives that position more weight in the calculation that follows. The values themselves are combined only after those weights have been computed.",[11,979,980,981,987],{},"For one head, the calculation introduced in ",[18,982,986],{"href":983,"rel":984},"https:\u002F\u002Farxiv.org\u002Fhtml\u002F1706.03762v7#S3.SS2.SSS1",[985],"nofollow","the Transformer paper"," is",[56,989,996,998,1085,1088,1090],{"style":258,"className":990,"viewBox":991,"width":992,"height":993,"dataWidth":994,"dataHeight":995,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],"0 0 118.000 33.000","10.727272727272727em","3em","118.000","33.000",[70,997,73],{"type":72},[75,999,1001,1004,1009,1014,1019,1024,1029,1034,1039,1044,1049,1054,1059,1064,1067,1072,1075,1080],{"className":1000},[78],[80,1002],{"id":272,"className":1003,"d":274},[84],[80,1005],{"id":1006,"className":1007,"d":1008},"gl3Hi1UI",[84],"M 483 91 L 483 150 L 451 150 L 451 91 C 451 52 440 32 419 32 C 399 32 387 57 387 77 L 387 274 C 387 309 383 335 376 353 C 352 412 283 448 213 448 C 136 448 60 405 60 333 C 60 300 77 283 110 283 C 143 283 159 299 159 332 C 159 361 144 378 113 381 C 135 406 168 419 211 419 C 273 419 311 362 311 297 L 311 264 C 233 259 174 247 133 228 C 66 197 32 154 32 97 C 32 71 42 49 61 32 C 93 3 137 -11 193 -11 C 252 -11 294 14 320 65 C 327 27 355 -6 398 -6 C 451 -6 483 36 483 91 Z M 200 18 C 154 18 116 52 116 98 C 116 191 213 231 311 236 L 311 141 C 311 74 267 18 200 18 Z ",[80,1010],{"id":1011,"className":1012,"d":1013},"gl3Hi1Uc",[84],"M 265 705 C 176 705 110 633 110 544 L 110 434 L 33 434 L 33 396 L 110 396 L 110 79 C 110 60 107 48 100 44 C 93 40 71 39 34 39 L 34 0 L 149 3 C 191 4 234 3 277 0 L 277 39 L 253 39 C 220 39 199 41 192 46 C 185 51 182 63 182 81 L 182 396 L 294 396 L 294 434 L 179 434 L 179 545 C 179 605 210 675 265 675 C 273 675 280 674 287 672 C 272 664 264 650 264 631 C 264 600 279 584 310 584 C 341 584 357 600 357 632 C 357 677 310 705 265 705 Z ",[80,1015],{"id":1016,"className":1017,"d":1018},"gl3Hi1U4",[84],"M 315 413 C 361 413 384 378 384 307 L 384 79 C 384 60 381 48 374 44 C 367 40 344 38 307 38 L 307 0 L 423 3 L 538 0 L 538 38 C 501 38 479 40 472 44 C 465 48 461 60 461 79 L 461 259 C 461 339 513 413 590 413 C 637 413 660 378 660 307 L 660 79 C 660 60 656 48 649 44 C 642 40 619 38 582 38 L 582 0 L 698 3 L 813 0 L 813 38 C 781 38 760 39 750 42 C 740 45 736 52 736 64 L 736 251 C 736 298 734 331 731 350 C 721 411 676 442 597 442 C 534 442 487 413 455 354 C 441 413 397 442 322 442 C 259 442 211 412 179 352 L 179 442 L 32 431 L 32 393 C 68 393 90 390 98 384 C 106 378 109 365 109 342 L 109 79 C 109 60 105 48 98 44 C 91 40 69 38 32 38 L 32 0 L 148 3 L 263 0 L 263 38 C 226 38 203 40 196 44 C 189 48 185 60 185 79 L 185 259 C 185 340 238 413 315 413 Z ",[80,1020],{"id":1021,"className":1022,"d":1023},"gl3Hi1VA",[84],"M 249 -11 C 311 -11 363 11 406 55 C 449 99 471 152 471 214 C 471 277 450 332 408 378 C 366 424 313 448 250 448 C 187 448 135 424 92 378 C 49 332 28 277 28 214 C 28 152 49 99 92 55 C 135 11 188 -11 249 -11 Z M 250 21 C 202 21 166 42 141 85 C 125 113 117 159 117 222 C 117 283 125 327 140 355 C 164 398 200 419 249 419 C 296 419 332 398 357 357 C 374 329 382 284 382 222 C 382 106 348 21 250 21 Z ",[80,1025],{"id":1026,"className":1027,"d":1028},"gl3Hi1VQ",[84],"M 360 130 C 360 206 308 254 205 274 C 158 283 128 294 112 305 C 96 316 88 332 88 351 C 88 398 123 421 193 421 C 261 421 298 383 303 306 C 304 298 309 294 319 294 C 330 294 336 303 336 321 L 336 420 C 336 439 331 448 321 448 C 310 448 295 430 285 422 C 260 439 229 448 193 448 C 104 448 33 408 33 323 C 33 289 47 260 76 237 C 111 208 144 202 207 191 C 272 178 305 149 305 104 C 305 47 270 18 199 18 C 130 18 86 64 67 155 C 64 168 59 175 50 175 C 39 175 33 165 33 146 L 33 17 C 33 -2 38 -11 48 -11 C 51 -11 56 -8 64 -1 C 82 15 74 12 92 30 C 121 3 157 -11 199 -11 C 294 -11 360 38 360 130 Z ",[80,1030],{"id":1031,"className":1032,"d":1033},"gl3Hi1VU",[84],"M 332 126 L 332 186 L 300 186 L 300 128 C 300 78 282 22 238 22 C 197 22 177 56 177 124 L 177 395 L 316 395 L 316 433 L 177 433 L 177 615 L 145 615 C 144 582 141 553 134 527 C 117 461 78 427 19 424 L 19 395 L 102 395 L 102 126 C 102 67 120 29 155 10 C 182 -4 207 -11 232 -11 C 298 -11 332 55 332 126 Z ",[80,1035],{"id":1036,"className":1037,"d":1038},"gl3Hi1Vk",[84],"M 418 3 C 441 4 474 3 516 0 L 516 38 C 463 38 443 39 422 67 L 292 235 C 335 288 364 324 380 343 C 408 375 447 392 498 393 L 498 431 C 470 429 442 428 414 428 C 377 428 344 429 315 431 L 315 393 C 334 391 344 382 344 366 C 344 357 339 346 330 334 L 272 262 L 198 357 C 193 364 190 369 190 372 C 190 385 202 392 225 393 L 225 431 L 114 428 C 88 427 56 428 17 431 L 17 393 C 53 393 76 390 86 385 C 96 380 112 363 134 334 L 229 210 C 180 147 150 109 139 96 C 107 57 65 38 12 38 L 12 0 C 49 3 77 4 98 3 L 195 0 L 195 38 C 176 41 167 50 167 65 C 167 69 168 74 170 79 C 176 92 203 126 251 183 L 323 89 C 334 75 340 65 343 59 C 343 46 331 39 307 38 L 307 0 Z ",[80,1040],{"id":1041,"className":1042,"d":1043},"gl3Hi1d0D",[84],"M 847 -2 C 851 7 853 13 853 16 C 853 32 845 40 829 40 C 820 40 813 36 809 27 C 666 -259 595 -403 595 -405 L 595 -406 C 595 -413 527 -558 390 -843 L 217 -461 C 212 -450 207 -445 201 -445 C 198 -445 192 -448 184 -454 L 86 -528 C 77 -535 73 -540 73 -545 C 73 -555 78 -560 87 -560 C 90 -560 95 -557 103 -551 L 151 -516 L 345 -943 C 350 -954 357 -960 367 -960 C 379 -960 387 -955 392 -946 Z ",[80,1045],{"id":1046,"className":1047,"d":1048},"gl3Hi1fgK",[84],"M 804 680 C 783 680 715 683 694 683 C 679 683 671 675 671 659 C 671 651 678 646 692 643 C 706 640 714 635 714 628 C 714 617 703 602 680 585 L 302 296 L 379 602 C 384 623 393 636 406 641 C 412 643 429 644 458 644 C 484 644 496 644 496 668 C 496 678 490 683 478 683 L 351 680 L 223 683 C 207 684 199 676 199 660 C 199 649 210 644 232 644 C 252 644 293 647 293 631 C 293 629 292 623 289 613 L 156 82 C 151 60 141 47 126 42 C 119 40 101 39 70 39 C 48 39 38 37 38 16 C 38 5 44 0 57 0 L 183 3 L 246 2 C 257 2 299 0 310 0 C 326 0 334 8 334 24 C 334 34 324 39 303 39 C 262 39 242 43 242 52 C 242 57 245 72 251 95 L 290 249 L 433 359 L 546 98 C 553 81 557 70 557 65 C 557 48 541 39 510 39 C 492 39 484 35 484 15 C 484 5 490 0 502 0 C 524 0 600 3 622 3 C 641 3 701 0 720 0 C 734 0 741 8 741 24 C 741 33 733 38 718 39 C 677 39 667 50 653 81 L 512 404 C 511 409 509 413 508 416 L 708 570 C 757 608 799 631 833 639 C 857 643 889 642 889 668 C 889 678 884 683 874 683 C 860 683 818 680 804 680 Z ",[80,1050],{"id":1051,"className":1052,"d":1053},"gl3Hi1QML",[84],"M 671 680 C 652 680 592 683 573 683 C 558 683 550 675 550 660 C 550 650 557 645 570 644 C 598 643 612 633 612 616 C 612 607 607 595 598 580 L 300 107 L 234 619 C 234 636 255 644 298 644 C 318 644 327 651 327 668 C 327 678 321 683 309 683 C 287 683 209 680 187 680 C 167 680 99 683 79 683 C 64 683 56 675 56 660 C 56 649 66 644 85 644 C 102 644 114 642 122 640 C 138 635 137 633 140 614 L 218 4 C 221 -13 229 -22 242 -22 C 255 -22 265 -15 273 -2 L 629 564 C 652 600 674 623 696 632 C 711 639 730 643 752 644 C 763 645 768 652 769 667 C 770 678 764 683 752 683 C 737 683 686 680 671 680 Z ",[80,1055],{"id":1056,"className":1057,"d":1058},"gl3Hi1QsL",[84],"M 429 632 L 370 389 C 349 426 320 445 281 445 C 216 445 159 412 109 345 C 63 283 40 218 40 151 C 40 63 91 -11 175 -11 C 218 -11 260 12 300 59 C 311 21 345 -11 392 -11 C 461 -11 483 71 498 145 C 498 154 493 159 483 159 C 474 159 468 152 465 138 C 445 59 421 19 394 19 C 377 19 368 33 368 60 C 368 75 370 91 374 107 L 516 679 L 516 683 C 513 690 507 694 499 694 C 476 694 434 690 374 683 C 363 682 357 674 357 660 C 357 650 366 645 384 645 C 403 645 429 646 429 632 Z M 341 376 C 351 355 356 341 356 332 C 355 328 354 323 353 316 L 304 122 C 301 111 294 99 285 87 C 248 42 212 19 177 19 C 138 19 118 49 118 108 C 118 132 124 168 136 216 C 157 301 187 359 224 390 C 244 407 263 415 282 415 C 309 415 329 402 341 376 Z ",[80,1060],{"id":1061,"className":1062,"d":1063},"gl3Hi1RUL",[84],"M 308 442 C 239 442 176 412 122 353 C 68 294 41 229 41 159 C 41 63 106 -11 202 -11 C 272 -11 334 19 388 78 C 442 137 469 201 469 272 C 469 369 405 442 308 442 Z M 389 310 C 389 287 384 255 374 213 C 353 130 319 73 272 42 C 248 26 225 18 203 18 C 149 18 121 65 121 122 C 121 180 155 288 178 324 C 216 383 259 413 307 413 C 361 413 389 367 389 310 Z ",[80,1065],{"id":290,"className":1066,"d":292},[84],[80,1068],{"id":1069,"className":1070,"d":1071},"gl3Hi1cUT",[84],"M 393 627 L 259 87 C 255 72 251 63 248 59 C 239 48 205 43 147 43 C 114 43 98 45 98 18 C 98 6 105 0 119 0 C 173 0 233 5 289 4 L 378 3 C 404 3 443 2 462 0 C 479 0 488 9 488 26 C 488 35 483 41 472 43 L 444 43 C 395 43 366 45 359 50 C 356 52 355 55 355 60 C 355 65 356 72 358 82 L 489 601 C 496 624 494 630 516 632 C 519 633 538 633 575 633 C 632 633 669 630 685 625 C 717 614 729 599 729 550 C 729 523 726 493 720 460 C 720 447 727 441 741 441 C 746 441 751 442 754 445 C 759 455 762 463 763 469 L 790 658 C 790 670 779 676 756 676 L 148 676 C 120 676 118 674 111 653 L 48 478 C 47 473 43 462 43 459 C 43 447 50 441 64 441 C 74 441 82 450 88 467 C 122 560 144 619 247 631 C 258 632 284 633 325 633 L 367 633 C 385 633 393 635 393 627 Z ",[80,1073],{"id":180,"className":1074,"d":182},[84],[80,1076],{"id":1077,"className":1078,"d":1079},"gl3Hi1fUW",[84],"M 798 -1245 C 815 -1245 823 -1237 823 -1220 C 823 -1212 820 -1205 813 -1200 C 732 -1139 659 -1043 592 -910 C 473 -673 385 -340 385 1 L 385 499 C 385 654 403 810 438 967 C 500 1247 630 1562 813 1700 C 820 1705 823 1712 823 1720 C 823 1737 815 1745 798 1745 C 793 1745 788 1743 784 1740 C 697 1673 614 1574 536 1441 C 398 1205 277 850 277 499 L 277 1 C 277 -329 388 -667 512 -899 C 594 -1050 685 -1164 784 -1240 C 788 -1243 793 -1245 798 -1245 Z ",[80,1081],{"id":1082,"className":1083,"d":1084},"gl3Hi1fYW",[84],"M 598 1 L 598 499 C 598 829 487 1167 363 1399 C 281 1550 190 1664 91 1740 C 87 1743 82 1745 77 1745 C 60 1745 52 1737 52 1720 C 52 1712 55 1705 62 1700 C 143 1639 216 1543 283 1410 C 402 1173 490 840 490 499 L 490 1 C 490 -154 472 -310 437 -467 C 375 -747 245 -1062 62 -1200 C 55 -1205 52 -1212 52 -1220 C 52 -1237 60 -1245 77 -1245 C 82 -1245 87 -1243 91 -1240 C 178 -1173 261 -1074 339 -941 C 477 -705 598 -350 598 1 Z ",[75,1086],{"className":1087},[89],[70,1089],{"type":72},[93,1091,1096],{"className":1092,"transform":97,"dataTid":1093,"dataPageWidth":1094,"dataPageHeight":1095},[96],"pPs+Cr191c4krzK9YK\u002F0Y6Q","118","33",[93,1097,1100],{"dataTid":1098,"className":1099},"gPs+Cr191c4krzK9YK\u002F0Y6Q",[105],[93,1101,1103],{"transform":108,"dataTid":1102},"pQl9LqBrc1CQllLgejVz1RI",[93,1104,1107,1415],{"dataTid":1105,"className":1106},"gQl9LqBrc1CQllLgejVz1RI",[105],[93,1108,1110],{"transform":108,"dataTid":1109},"p246eDJDfptEiIJKqKHWdLE",[93,1111,1114,1134,1149,1164,1208,1228,1372,1391,1410],{"dataTid":1112,"className":1113},"g246eDJDfptEiIJKqKHWdLE",[105],[93,1115,1118],{"transform":1116,"dataTid":1117},"translate(0.000,19.195)","pyE\u002Fh24cQ19w4\u002FZyQr3M8m8",[93,1119,1122,1125],{"dataTid":1120,"className":1121,"fill":130,"transform":131},"gyE\u002Fh24cQ19w4\u002FZyQr3M8m8",[129],[133,1123],{"x":135,"y":135,"href":1124},"#gl3Hi1RUL",[93,1126,1127],{"transform":139},[141,1128,1130],{"x":135,"y":143,"width":1129,"height":145},"31.06",[147,1131,1133],{"className":1132,"style":151},[150],"𝑜",[93,1135,1137],{"transform":1136,"dataTid":233},"translate(5.335,21.912)",[93,1138,1140,1142],{"dataTid":236,"className":1139,"fill":130,"transform":238},[129],[133,1141],{"x":135,"y":135,"href":241},[93,1143,1144],{"transform":139},[141,1145,1146],{"x":135,"y":143,"width":246,"height":145},[147,1147,152],{"className":1148,"style":151},[150],[93,1150,1152],{"transform":1151,"dataTid":384},"translate(12.333,19.195)",[93,1153,1155,1157],{"dataTid":387,"className":1154,"fill":130,"transform":131},[129],[133,1156],{"x":135,"y":135,"href":391},[93,1158,1159],{"transform":139},[141,1160,1161],{"x":135,"y":143,"width":396,"height":145},[147,1162,400],{"className":1163,"style":151},[150],[93,1165,1168],{"transform":1166,"dataTid":1167},"translate(23.947,19.195)","pO+eGEwODvRU7lWYl+jAkXo",[93,1169,1172,1175,1179,1183,1187,1191,1195,1199],{"dataTid":1170,"className":1171,"fill":130,"transform":131},"gO+eGEwODvRU7lWYl+jAkXo",[129],[133,1173],{"x":135,"y":135,"href":1174},"#gl3Hi1VQ",[133,1176],{"x":1177,"y":135,"href":1178},"394","#gl3Hi1VA",[133,1180],{"x":1181,"y":135,"href":1182},"894","#gl3Hi1Uc",[133,1184],{"x":1185,"y":135,"href":1186},"1200","#gl3Hi1VU",[133,1188],{"x":1189,"y":135,"href":1190},"1589","#gl3Hi1U4",[133,1192],{"x":1193,"y":135,"href":1194},"2422","#gl3Hi1UI",[133,1196],{"x":1197,"y":135,"href":1198},"2922","#gl3Hi1Vk",[93,1200,1201],{"transform":139},[141,1202,1204],{"x":135,"y":143,"width":1203,"height":145},"215.62",[147,1205,1207],{"className":1206,"style":151},[150],"softmax",[93,1209,1212],{"transform":1210,"dataTid":1211},"translate(61.897,19.195)","poEsihrMsCDQzkCW+ZETc8E",[93,1213,1216,1219],{"dataTid":1214,"className":1215,"fill":130,"transform":131},"goEsihrMsCDQzkCW+ZETc8E",[129],[133,1217],{"x":135,"y":135,"href":1218},"#gl3Hi1fUW",[93,1220,1221],{"transform":139},[141,1222,1224],{"x":135,"y":143,"width":1223,"height":145},"54.69",[147,1225,1227],{"className":1226,"style":151},[150],"(",[93,1229,1232],{"transform":1230,"dataTid":1231},"translate(71.522,2.550)","pKy1yPJqZii4x1MWg\u002F72O5M",[93,1233,1236,1251,1266,1285,1305,1325,1341,1361],{"dataTid":1234,"className":1235},"gKy1yPJqZii4x1MWg\u002F72O5M",[105],[93,1237,1239],{"transform":1238,"dataTid":349},"translate(1.100,9.198)",[93,1240,1242,1244],{"dataTid":352,"className":1241,"fill":130,"transform":131},[129],[133,1243],{"x":135,"y":135,"href":356},[93,1245,1246],{"transform":139},[141,1247,1248],{"x":135,"y":143,"width":361,"height":145},[147,1249,365],{"className":1250,"style":151},[150],[93,1252,1254],{"transform":1253,"dataTid":233},"translate(6.006,11.915)",[93,1255,1257,1259],{"dataTid":236,"className":1256,"fill":130,"transform":238},[129],[133,1258],{"x":135,"y":135,"href":241},[93,1260,1261],{"transform":139},[141,1262,1263],{"x":135,"y":143,"width":246,"height":145},[147,1264,152],{"className":1265,"style":151},[150],[93,1267,1270],{"transform":1268,"dataTid":1269},"translate(9.948,9.198)","pjXYRl9+L0EEolYug2DunfY",[93,1271,1274,1277],{"dataTid":1272,"className":1273,"fill":130,"transform":131},"gjXYRl9+L0EEolYug2DunfY",[129],[133,1275],{"x":135,"y":135,"href":1276},"#gl3Hi1fgK",[93,1278,1279],{"transform":139},[141,1280,1282],{"x":135,"y":143,"width":1281,"height":145},"57.31",[147,1283,597],{"className":1284,"style":151},[150],[93,1286,1289],{"transform":1287,"dataTid":1288},"translate(20.035,5.205)","p45gwfYPR5FIiAOQYgbFw7o",[93,1290,1293,1296],{"dataTid":1291,"className":1292,"fill":130,"transform":238},"g45gwfYPR5FIiAOQYgbFw7o",[129],[133,1294],{"x":135,"y":135,"href":1295},"#gl3Hi1cUT",[93,1297,1298],{"transform":139},[141,1299,1301],{"x":135,"y":143,"width":1300,"height":145},"51.13",[147,1302,1304],{"className":1303,"style":151},[150],"𝑇",[93,1306,1309],{"transform":1307,"dataTid":1308},"translate(6.451,16.447)","p6lLbSRnN66w7vSMd6ws\u002FuY",[93,1310,1313,1316],{"dataTid":1311,"className":1312,"fill":130,"transform":131},"g6lLbSRnN66w7vSMd6ws\u002FuY",[129],[133,1314],{"x":135,"y":135,"href":1315},"#gl3Hi1d0D",[93,1317,1318],{"transform":139},[141,1319,1321],{"x":135,"y":143,"width":1320,"height":145},"52.06",[147,1322,1324],{"className":1323,"style":151},[150],"√",[93,1326,1329],{"transform":1327,"dataTid":1328},"translate(15.615,16.271)","pZIrRezOugfIMPEmptF3G5M",[93,1330,1332],{"dataTid":1331},"gZIrRezOugfIMPEmptF3G5M",[80,1333],{"className":1334,"d":1336,"strokeWidth":1337,"strokeMiterLimit":1338,"fillRule":1339,"fill":1340,"stroke":130},[1335],"typst-shape","M 0 0 L 5.984 0 ","0.528",4,"nonzero","none",[93,1342,1345],{"transform":1343,"dataTid":1344},"translate(15.615,25.803)","pcs7iuBS0TA4jxSbb6T+Sz4",[93,1346,1349,1352],{"dataTid":1347,"className":1348,"fill":130,"transform":131},"gcs7iuBS0TA4jxSbb6T+Sz4",[129],[133,1350],{"x":135,"y":135,"href":1351},"#gl3Hi1QsL",[93,1353,1354],{"transform":139},[141,1355,1357],{"x":135,"y":143,"width":1356,"height":145},"34.00",[147,1358,1360],{"className":1359,"style":151},[150],"𝑑",[93,1362,1365],{"transform":1363,"dataTid":1364},"translate(1.100,13.895)","peCzIc7VslEgYPWFVn4NFb8",[93,1366,1368],{"dataTid":1367},"geCzIc7VslEgYPWFVn4NFb8",[80,1369],{"className":1370,"d":1371,"strokeWidth":1337,"strokeMiterLimit":1338,"fillRule":1339,"fill":1340,"stroke":130},[1335],"M 0 0 L 25.85 0 ",[93,1373,1376],{"transform":1374,"dataTid":1375},"translate(99.572,19.195)","pbymWxyxE24cANafQCBqh4o",[93,1377,1380,1383],{"dataTid":1378,"className":1379,"fill":130,"transform":131},"gbymWxyxE24cANafQCBqh4o",[129],[133,1381],{"x":135,"y":135,"href":1382},"#gl3Hi1fYW",[93,1384,1385],{"transform":139},[141,1386,1387],{"x":135,"y":143,"width":1223,"height":145},[147,1388,1390],{"className":1389,"style":151},[150],")",[93,1392,1395],{"transform":1393,"dataTid":1394},"translate(109.197,19.195)","p9ivEDomtzaU3Ug0oyx+0kI",[93,1396,1399,1402],{"dataTid":1397,"className":1398,"fill":130,"transform":131},"g9ivEDomtzaU3Ug0oyx+0kI",[129],[133,1400],{"x":135,"y":135,"href":1401},"#gl3Hi1QML",[93,1403,1404],{"transform":139},[141,1405,1407],{"x":135,"y":143,"width":1406,"height":145},"49.81",[147,1408,724],{"className":1409,"style":151},[150],[93,1411,1412],{"transform":108,"dataTid":155},[93,1413],{"dataTid":158,"className":1414,"dataHint":18},[160],[93,1416,1417],{"transform":108,"dataTid":155},[93,1418],{"dataTid":158,"className":1419,"dataHint":18},[160],[11,1421,1422,1423,1502,1503,1554,1555,1618,1619,1680],{},"Here ",[56,1424,1429,1431,1440,1443,1445],{"style":58,"className":1425,"viewBox":1426,"width":1427,"height":63,"dataWidth":1428,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],"0 0 9.000 19.000","0.8181818181818182em","9.000",[70,1430,73],{"type":72},[75,1432,1434,1437],{"className":1433},[78],[80,1435],{"id":290,"className":1436,"d":292},[84],[80,1438],{"id":180,"className":1439,"d":182},[84],[75,1441],{"className":1442},[89],[70,1444],{"type":72},[93,1446,1450],{"className":1447,"transform":97,"dataTid":1448,"dataPageWidth":1449,"dataPageHeight":100},[96],"p9Q7X7DM1gzEOsZJyADVxmw","9",[93,1451,1454],{"dataTid":1452,"className":1453},"g9Q7X7DM1gzEOsZJyADVxmw",[105],[93,1455,1457],{"transform":108,"dataTid":1456},"pIAy1olANLV07Odj3vvBCls",[93,1458,1461,1497],{"dataTid":1459,"className":1460},"gIAy1olANLV07Odj3vvBCls",[105],[93,1462,1464],{"transform":108,"dataTid":1463},"p\u002FL39a\u002FTbBRsXmNHT2aBgCw",[93,1465,1468,1482],{"dataTid":1466,"className":1467},"g\u002FL39a\u002FTbBRsXmNHT2aBgCw",[105],[93,1469,1470],{"transform":123,"dataTid":349},[93,1471,1473,1475],{"dataTid":352,"className":1472,"fill":130,"transform":131},[129],[133,1474],{"x":135,"y":135,"href":356},[93,1476,1477],{"transform":139},[141,1478,1479],{"x":135,"y":143,"width":361,"height":145},[147,1480,365],{"className":1481,"style":151},[150],[93,1483,1485],{"transform":1484,"dataTid":233},"translate(4.906,15.730)",[93,1486,1488,1490],{"dataTid":236,"className":1487,"fill":130,"transform":238},[129],[133,1489],{"x":135,"y":135,"href":241},[93,1491,1492],{"transform":139},[141,1493,1494],{"x":135,"y":143,"width":246,"height":145},[147,1495,152],{"className":1496,"style":151},[150],[93,1498,1499],{"transform":108,"dataTid":155},[93,1500],{"dataTid":158,"className":1501,"dataHint":18},[160]," is the current query. K and V contain the keys and values for positions 1 through ",[56,1504,1506,1508,1514,1517,1519],{"style":58,"className":1505,"viewBox":61,"width":62,"height":63,"dataWidth":64,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],[70,1507,73],{"type":72},[75,1509,1511],{"className":1510},[78],[80,1512],{"id":82,"className":1513,"d":85},[84],[75,1515],{"className":1516},[89],[70,1518],{"type":72},[93,1520,1522],{"className":1521,"transform":97,"dataTid":98,"dataPageWidth":99,"dataPageHeight":100},[96],[93,1523,1525],{"dataTid":103,"className":1524},[105],[93,1526,1527],{"transform":108,"dataTid":109},[93,1528,1530,1549],{"dataTid":112,"className":1529},[105],[93,1531,1532],{"transform":108,"dataTid":116},[93,1533,1535],{"dataTid":119,"className":1534},[105],[93,1536,1537],{"transform":123,"dataTid":124},[93,1538,1540,1542],{"dataTid":127,"className":1539,"fill":130,"transform":131},[129],[133,1541],{"x":135,"y":135,"href":136},[93,1543,1544],{"transform":139},[141,1545,1546],{"x":135,"y":143,"width":144,"height":145},[147,1547,152],{"className":1548,"style":151},[150],[93,1550,1551],{"transform":108,"dataTid":155},[93,1552],{"dataTid":158,"className":1553,"dataHint":18},[160],", so the current position is included and future positions are excluded. The symbol ",[56,1556,1558,1560,1568,1571,1573],{"style":58,"className":1557,"viewBox":1426,"width":1427,"height":63,"dataWidth":1428,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],[70,1559,73],{"type":72},[75,1561,1563],{"className":1562},[78],[80,1564],{"id":1565,"className":1566,"d":1567},"gl3Hi1QEL",[84],"M 344 631 C 344 628 343 621 340 611 L 208 83 C 204 68 200 59 197 55 C 188 44 154 39 94 39 C 63 39 49 42 49 16 C 49 5 56 0 69 0 C 120 0 192 4 235 3 L 317 2 C 331 2 386 0 403 0 C 420 0 428 8 428 24 C 428 34 416 39 391 39 C 339 39 309 41 300 46 C 297 48 295 52 295 58 L 430 603 C 433 617 436 626 439 629 C 443 635 467 638 511 638 C 566 638 604 634 623 625 C 642 616 652 595 652 561 C 652 543 649 517 644 483 C 642 475 641 469 641 464 C 641 453 646 447 657 447 C 666 447 672 456 675 473 L 702 645 C 703 650 704 656 704 662 C 704 672 694 677 673 677 L 125 677 C 99 677 97 674 89 655 L 30 481 C 27 472 25 466 24 462 C 24 452 29 447 40 447 C 48 447 55 455 60 470 C 85 543 110 588 133 607 C 159 628 209 638 282 638 L 321 638 C 332 638 344 639 344 631 Z ",[75,1569],{"className":1570},[89],[70,1572],{"type":72},[93,1574,1577],{"className":1575,"transform":97,"dataTid":1576,"dataPageWidth":1449,"dataPageHeight":100},[96],"p93VnvQlpAdgQ5bsP4eSd+A",[93,1578,1581],{"dataTid":1579,"className":1580},"g93VnvQlpAdgQ5bsP4eSd+A",[105],[93,1582,1584],{"transform":108,"dataTid":1583},"pi1FKGsQF1SMyNBQvUl0fUs",[93,1585,1588,1613],{"dataTid":1586,"className":1587},"gi1FKGsQF1SMyNBQvUl0fUs",[105],[93,1589,1591],{"transform":108,"dataTid":1590},"pvzx5SLVImB0tsPPGODtDiY",[93,1592,1595],{"dataTid":1593,"className":1594},"gvzx5SLVImB0tsPPGODtDiY",[105],[93,1596,1598],{"transform":123,"dataTid":1597},"pj2XrWSyS9HAjlxkkLKHbr8",[93,1599,1602,1605],{"dataTid":1600,"className":1601,"fill":130,"transform":131},"gj2XrWSyS9HAjlxkkLKHbr8",[129],[133,1603],{"x":135,"y":135,"href":1604},"#gl3Hi1QEL",[93,1606,1607],{"transform":139},[141,1608,1610],{"x":135,"y":143,"width":1609,"height":145},"45.75",[147,1611,1304],{"className":1612,"style":151},[150],[93,1614,1615],{"transform":108,"dataTid":155},[93,1616],{"dataTid":158,"className":1617,"dataHint":18},[160]," transposes K, lining up its keys for the dot products. The dimension ",[56,1620,1625,1627,1633,1636,1638],{"style":58,"className":1621,"viewBox":1622,"width":1623,"height":63,"dataWidth":1624,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],"0 0 6.000 19.000","0.5454545454545454em","6.000",[70,1626,73],{"type":72},[75,1628,1630],{"className":1629},[78],[80,1631],{"id":1056,"className":1632,"d":1058},[84],[75,1634],{"className":1635},[89],[70,1637],{"type":72},[93,1639,1643],{"className":1640,"transform":97,"dataTid":1641,"dataPageWidth":1642,"dataPageHeight":100},[96],"pKXntX3eORqY4zw60cTGzx0","6",[93,1644,1647],{"dataTid":1645,"className":1646},"gKXntX3eORqY4zw60cTGzx0",[105],[93,1648,1650],{"transform":108,"dataTid":1649},"pRPU6uo1j8\u002F4ZUVGHAzrOsk",[93,1651,1654,1675],{"dataTid":1652,"className":1653},"gRPU6uo1j8\u002F4ZUVGHAzrOsk",[105],[93,1655,1657],{"transform":108,"dataTid":1656},"pACdq9peIy6A2K3JwGEDDNo",[93,1658,1661],{"dataTid":1659,"className":1660},"gACdq9peIy6A2K3JwGEDDNo",[105],[93,1662,1663],{"transform":123,"dataTid":1344},[93,1664,1666,1668],{"dataTid":1347,"className":1665,"fill":130,"transform":131},[129],[133,1667],{"x":135,"y":135,"href":1351},[93,1669,1670],{"transform":139},[141,1671,1672],{"x":135,"y":143,"width":1356,"height":145},[147,1673,1360],{"className":1674,"style":151},[150],[93,1676,1677],{"transform":108,"dataTid":155},[93,1678],{"dataTid":158,"className":1679,"dataHint":18},[160]," is the number of components in each query and key vector for this head.",[11,1682,1683,1684,1798,1799,1887,1888,1967],{},"The product ",[56,1685,1690,1692,1707,1710,1712],{"style":58,"className":1686,"viewBox":1687,"width":1688,"height":63,"dataWidth":1689,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],"0 0 26.000 19.000","2.3636363636363638em","26.000",[70,1691,73],{"type":72},[75,1693,1695,1698,1701,1704],{"className":1694},[78],[80,1696],{"id":1046,"className":1697,"d":1048},[84],[80,1699],{"id":290,"className":1700,"d":292},[84],[80,1702],{"id":1069,"className":1703,"d":1071},[84],[80,1705],{"id":180,"className":1706,"d":182},[84],[75,1708],{"className":1709},[89],[70,1711],{"type":72},[93,1713,1717],{"className":1714,"transform":97,"dataTid":1715,"dataPageWidth":1716,"dataPageHeight":100},[96],"pVhw7II2IH+YPxxwn8ghAjE","26",[93,1718,1721],{"dataTid":1719,"className":1720},"gVhw7II2IH+YPxxwn8ghAjE",[105],[93,1722,1724],{"transform":108,"dataTid":1723},"pYBftuSN8t\u002F8muLYbAyxet8",[93,1725,1728,1793],{"dataTid":1726,"className":1727},"gYBftuSN8t\u002F8muLYbAyxet8",[105],[93,1729,1731],{"transform":108,"dataTid":1730},"pN\u002F9WqVXmBQIOrwA9aMKzkY",[93,1732,1735,1749,1763,1778],{"dataTid":1733,"className":1734},"gN\u002F9WqVXmBQIOrwA9aMKzkY",[105],[93,1736,1737],{"transform":123,"dataTid":349},[93,1738,1740,1742],{"dataTid":352,"className":1739,"fill":130,"transform":131},[129],[133,1741],{"x":135,"y":135,"href":356},[93,1743,1744],{"transform":139},[141,1745,1746],{"x":135,"y":143,"width":361,"height":145},[147,1747,365],{"className":1748,"style":151},[150],[93,1750,1751],{"transform":1484,"dataTid":233},[93,1752,1754,1756],{"dataTid":236,"className":1753,"fill":130,"transform":238},[129],[133,1755],{"x":135,"y":135,"href":241},[93,1757,1758],{"transform":139},[141,1759,1760],{"x":135,"y":143,"width":246,"height":145},[147,1761,152],{"className":1762,"style":151},[150],[93,1764,1766],{"transform":1765,"dataTid":1269},"translate(8.848,13.013)",[93,1767,1769,1771],{"dataTid":1272,"className":1768,"fill":130,"transform":131},[129],[133,1770],{"x":135,"y":135,"href":1276},[93,1772,1773],{"transform":139},[141,1774,1775],{"x":135,"y":143,"width":1281,"height":145},[147,1776,597],{"className":1777,"style":151},[150],[93,1779,1781],{"transform":1780,"dataTid":1288},"translate(18.935,9.020)",[93,1782,1784,1786],{"dataTid":1291,"className":1783,"fill":130,"transform":238},[129],[133,1785],{"x":135,"y":135,"href":1295},[93,1787,1788],{"transform":139},[141,1789,1790],{"x":135,"y":143,"width":1300,"height":145},[147,1791,1304],{"className":1792,"style":151},[150],[93,1794,1795],{"transform":108,"dataTid":155},[93,1796],{"dataTid":158,"className":1797,"dataHint":18},[160]," gives the scores. Dividing by ",[56,1800,1805,1807,1816,1819,1821],{"style":58,"className":1801,"viewBox":1802,"width":1803,"height":63,"dataWidth":1804,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],"0 0 16.000 19.000","1.4545454545454546em","16.000",[70,1806,73],{"type":72},[75,1808,1810,1813],{"className":1809},[78],[80,1811],{"id":1041,"className":1812,"d":1043},[84],[80,1814],{"id":1056,"className":1815,"d":1058},[84],[75,1817],{"className":1818},[89],[70,1820],{"type":72},[93,1822,1826],{"className":1823,"transform":97,"dataTid":1824,"dataPageWidth":1825,"dataPageHeight":100},[96],"pIofUKfislz4eCkFL6HnrNY","16",[93,1827,1830],{"dataTid":1828,"className":1829},"gIofUKfislz4eCkFL6HnrNY",[105],[93,1831,1833],{"transform":108,"dataTid":1832},"pT88N1ZHxuXoi8M3cqoLk4w",[93,1834,1837,1882],{"dataTid":1835,"className":1836},"gT88N1ZHxuXoi8M3cqoLk4w",[105],[93,1838,1840],{"transform":108,"dataTid":1839},"p\u002F8jXSdzbEssV+xcIkWnYuE",[93,1841,1844,1859,1867],{"dataTid":1842,"className":1843},"g\u002F8jXSdzbEssV+xcIkWnYuE",[105],[93,1845,1847],{"transform":1846,"dataTid":1308},"translate(0.000,3.658)",[93,1848,1850,1852],{"dataTid":1311,"className":1849,"fill":130,"transform":131},[129],[133,1851],{"x":135,"y":135,"href":1315},[93,1853,1854],{"transform":139},[141,1855,1856],{"x":135,"y":143,"width":1320,"height":145},[147,1857,1324],{"className":1858,"style":151},[150],[93,1860,1862],{"transform":1861,"dataTid":1328},"translate(9.163,3.481)",[93,1863,1864],{"dataTid":1331},[80,1865],{"className":1866,"d":1336,"strokeWidth":1337,"strokeMiterLimit":1338,"fillRule":1339,"fill":1340,"stroke":130},[1335],[93,1868,1870],{"transform":1869,"dataTid":1344},"translate(9.163,13.013)",[93,1871,1873,1875],{"dataTid":1347,"className":1872,"fill":130,"transform":131},[129],[133,1874],{"x":135,"y":135,"href":1351},[93,1876,1877],{"transform":139},[141,1878,1879],{"x":135,"y":143,"width":1356,"height":145},[147,1880,1360],{"className":1881,"style":151},[150],[93,1883,1884],{"transform":108,"dataTid":155},[93,1885],{"dataTid":158,"className":1886,"dataHint":18},[160]," keeps their size under control as the vectors get wider. Softmax then turns the scores into positive weights that add up to one. Multiplying those weights by V forms a weighted sum of the value vectors. That sum is the head's output ",[56,1889,1894,1896,1905,1908,1910],{"style":58,"className":1890,"viewBox":1891,"width":1892,"height":63,"dataWidth":1893,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],"0 0 10.000 19.000","0.9090909090909091em","10.000",[70,1895,73],{"type":72},[75,1897,1899,1902],{"className":1898},[78],[80,1900],{"id":1061,"className":1901,"d":1063},[84],[80,1903],{"id":180,"className":1904,"d":182},[84],[75,1906],{"className":1907},[89],[70,1909],{"type":72},[93,1911,1915],{"className":1912,"transform":97,"dataTid":1913,"dataPageWidth":1914,"dataPageHeight":100},[96],"pPfk\u002Fgwe3Zxw85+VmHwGFR0","10",[93,1916,1919],{"dataTid":1917,"className":1918},"gPfk\u002Fgwe3Zxw85+VmHwGFR0",[105],[93,1920,1922],{"transform":108,"dataTid":1921},"psmUcxS6YXDwGOp+9cykjdE",[93,1923,1926,1962],{"dataTid":1924,"className":1925},"gsmUcxS6YXDwGOp+9cykjdE",[105],[93,1927,1929],{"transform":108,"dataTid":1928},"pJ09E+6fECY8xnHKYOZRs+Y",[93,1930,1933,1947],{"dataTid":1931,"className":1932},"gJ09E+6fECY8xnHKYOZRs+Y",[105],[93,1934,1935],{"transform":123,"dataTid":1117},[93,1936,1938,1940],{"dataTid":1120,"className":1937,"fill":130,"transform":131},[129],[133,1939],{"x":135,"y":135,"href":1124},[93,1941,1942],{"transform":139},[141,1943,1944],{"x":135,"y":143,"width":1129,"height":145},[147,1945,1133],{"className":1946,"style":151},[150],[93,1948,1950],{"transform":1949,"dataTid":233},"translate(5.335,15.730)",[93,1951,1953,1955],{"dataTid":236,"className":1952,"fill":130,"transform":238},[129],[133,1954],{"x":135,"y":135,"href":241},[93,1956,1957],{"transform":139},[141,1958,1959],{"x":135,"y":143,"width":246,"height":145},[147,1960,152],{"className":1961,"style":151},[150],[93,1963,1964],{"transform":108,"dataTid":155},[93,1965],{"dataTid":158,"className":1966,"dataHint":18},[160],".",[11,1969,1970,1971,896,1975,1978],{},"For a small illustrative example, suppose there are two value vectors, ",[1972,1973,1974],"code",{},"(2, 0)",[1972,1976,1977],{},"(0, 4)",", and attention gives them weights of 0.25 and 0.75. The weighted sum is",[56,1980,1985,1987,2053,2056,2058],{"style":258,"className":1981,"viewBox":1982,"width":1983,"height":166,"dataWidth":1984,"dataHeight":167,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],"0 0 177.000 11.000","16.09090909090909em","177.000",[70,1986,73],{"type":72},[75,1988,1990,1995,2000,2005,2010,2015,2020,2025,2030,2035,2040,2045,2048],{"className":1989},[78],[80,1991],{"id":1992,"className":1993,"d":1994},"gl3Hi1Qk",[84],"M 318 -248 C 327 -248 332 -243 332 -234 C 332 -231 330 -227 327 -223 C 275 -183 233 -117 202 -26 C 175 53 161 131 161 208 L 161 292 C 161 369 175 447 202 526 C 233 617 275 683 327 723 C 330 726 332 730 332 734 C 332 743 327 748 318 748 C 317 748 314 747 311 745 C 251 699 201 631 160 540 C 121 453 101 371 101 292 L 101 208 C 101 129 121 47 160 -40 C 201 -131 251 -199 311 -245 C 314 -247 317 -248 318 -248 Z ",[80,1996],{"id":1997,"className":1998,"d":1999},"gl3Hi1Qo",[84],"M 78 -245 C 138 -199 188 -131 229 -40 C 268 47 288 129 288 208 L 288 292 C 288 371 268 453 229 540 C 188 631 138 699 78 745 C 75 747 72 748 71 748 C 62 748 57 743 57 734 C 57 730 59 726 62 723 C 114 683 156 617 187 526 C 214 447 228 369 228 292 L 228 208 C 228 131 214 53 187 -26 C 156 -117 114 -183 62 -223 C 59 -227 57 -231 57 -234 C 57 -243 62 -248 71 -248 C 72 -248 75 -247 78 -245 Z ",[80,2001],{"id":2002,"className":2003,"d":2004},"gl3Hi1Qw",[84],"M 698 274 L 413 274 L 413 559 C 413 575 405 583 389 583 C 373 583 365 575 365 559 L 365 274 L 80 274 C 64 274 56 266 56 250 C 56 234 64 226 80 226 L 365 226 L 365 -59 C 365 -75 373 -83 389 -83 C 405 -83 413 -75 413 -59 L 413 226 L 698 226 C 714 226 722 234 722 250 C 722 263 711 274 698 274 Z ",[80,2006],{"id":2007,"className":2008,"d":2009},"gl3Hi1Q0",[84],"M 139 106 C 107 106 86 82 86 50 C 86 20 109 -5 139 -5 C 153 -5 165 -1 174 8 L 175 0 C 175 -63 154 -117 112 -160 C 105 -168 101 -174 101 -178 C 101 -188 105 -193 114 -193 C 123 -193 135 -181 152 -158 C 186 -110 203 -57 203 0 C 203 53 185 106 139 106 Z ",[80,2011],{"id":2012,"className":2013,"d":2014},"gl3Hi1Q8",[84],"M 192 53 C 192 82 168 106 139 106 C 110 106 86 82 86 53 C 86 24 110 0 139 0 C 168 0 192 24 192 53 Z ",[80,2016],{"id":2017,"className":2018,"d":2019},"gl3Hi1RE",[84],"M 249 -22 C 390 -22 460 92 460 320 C 460 473 428 575 365 625 C 330 652 291 666 250 666 C 109 666 39 551 39 320 C 39 136 88 -22 249 -22 Z M 361 524 C 368 489 371 425 371 332 C 371 240 367 172 360 128 C 347 48 310 8 249 8 C 226 8 203 17 182 34 C 155 57 139 104 132 176 C 129 201 128 253 128 332 C 128 419 131 480 136 513 C 145 568 163 603 191 618 C 213 630 232 636 249 636 C 314 636 350 583 361 524 Z ",[80,2021],{"id":2022,"className":2023,"d":2024},"gl3Hi1RM",[84],"M 237 666 C 186 666 143 648 106 612 C 69 576 50 534 50 483 C 50 449 75 424 106 424 C 136 424 161 450 161 480 C 161 513 137 536 105 536 C 102 536 100 536 98 535 C 117 584 161 627 224 627 C 306 627 352 556 352 470 C 352 403 318 331 250 255 L 62 43 C 49 28 50 29 50 0 L 421 0 L 450 180 L 417 180 C 409 129 402 100 396 91 C 391 86 361 84 306 84 L 139 84 L 236 179 C 304 243 390 312 419 365 C 439 400 449 435 449 470 C 449 588 357 666 237 666 Z ",[80,2026],{"id":2027,"className":2028,"d":2029},"gl3Hi1RQ",[84],"M 303 353 C 369 378 431 441 431 526 C 431 569 410 604 369 631 C 333 654 292 666 246 666 C 201 666 162 654 127 631 C 88 605 68 571 68 528 C 68 495 90 472 122 472 C 154 472 176 495 176 527 C 176 560 157 578 119 580 C 145 615 186 633 242 633 C 302 633 332 598 332 527 C 332 485 324 450 309 421 C 282 373 245 364 183 364 C 171 362 165 357 165 348 C 165 333 172 333 192 333 L 235 333 C 310 333 348 280 348 173 C 348 88 317 14 241 14 C 176 14 128 36 99 80 C 134 79 160 105 160 139 C 160 173 135 198 101 198 C 62 198 42 178 42 137 C 42 88 64 49 108 18 C 147 -9 193 -22 244 -22 C 301 -22 350 -3 393 34 C 436 71 457 117 457 173 C 457 267 383 332 303 353 Z ",[80,2031],{"id":2032,"className":2033,"d":2034},"gl3Hi1RU",[84],"M 353 677 C 344 677 336 672 330 663 L 28 199 L 28 163 L 289 163 L 289 81 C 289 63 285 51 278 46 C 271 41 252 39 219 39 L 194 39 L 194 0 C 223 2 269 3 331 3 C 393 3 439 2 468 0 L 468 39 L 443 39 C 410 39 391 41 384 46 C 377 51 373 63 373 81 L 373 163 L 471 163 L 471 202 L 373 202 L 373 660 C 373 670 366 677 353 677 Z M 295 553 L 295 202 L 67 202 Z ",[80,2036],{"id":2037,"className":2038,"d":2039},"gl3Hi1RY",[84],"M 118 315 C 123 315 129 319 134 326 C 164 371 205 393 257 393 C 292 393 319 373 337 332 C 348 305 354 264 354 209 C 354 146 346 102 331 76 C 306 35 272 14 229 14 C 162 14 109 62 91 114 C 94 113 96 114 100 114 C 130 114 155 137 155 167 C 155 198 130 219 100 219 C 65 219 50 200 50 163 C 50 63 131 -22 231 -22 C 292 -22 344 0 386 44 C 428 88 449 141 449 202 C 449 260 432 310 398 353 C 361 400 315 423 259 423 C 212 423 171 408 138 378 L 138 556 C 165 548 191 544 218 544 C 264 544 304 555 338 578 C 369 597 390 616 402 634 C 408 642 411 648 411 651 C 411 661 406 666 396 666 C 341 645 294 634 256 634 C 211 634 168 643 127 662 C 122 664 118 665 114 665 C 105 665 100 656 100 637 L 100 345 C 100 324 100 315 118 315 Z ",[80,2041],{"id":2042,"className":2043,"d":2044},"gl3Hi1Rg",[84],"M 475 604 C 482 613 485 626 485 644 L 243 644 C 174 644 135 648 128 657 C 125 660 122 667 120 676 L 89 676 L 55 464 L 88 464 C 98 520 106 550 112 555 C 115 558 146 560 205 560 L 401 560 L 295 410 C 214 295 174 171 174 36 C 174 -3 190 -22 223 -22 C 256 -22 272 -3 272 36 L 272 87 C 272 239 296 349 343 416 Z ",[80,2046],{"id":272,"className":2047,"d":274},[84],[80,2049],{"id":2050,"className":2051,"d":2052},"gl3Hi1Zg",[84],"M 630 32 C 630 39 628 44 623 49 L 422 250 L 623 451 C 628 456 630 461 630 468 C 630 481 620 491 607 491 C 600 491 595 489 590 484 L 389 283 L 188 484 C 183 489 178 491 171 491 C 158 491 148 481 148 468 C 148 461 150 456 155 451 L 356 250 L 155 49 C 150 44 148 39 148 32 C 148 19 158 9 171 9 C 178 9 183 11 188 16 L 389 217 L 590 16 C 595 11 600 9 607 9 C 620 9 630 19 630 32 Z ",[75,2054],{"className":2055},[89],[70,2057],{"type":72},[93,2059,2063],{"className":2060,"transform":97,"dataTid":2061,"dataPageWidth":2062,"dataPageHeight":192},[96],"psf+5Rduqg9UamqTmjHEP\u002Fs","177",[93,2064,2067],{"dataTid":2065,"className":2066},"gsf+5Rduqg9UamqTmjHEP\u002Fs",[105],[93,2068,2070],{"transform":108,"dataTid":2069},"prU8\u002FiN1f8bgeG2W\u002F3+Sktw",[93,2071,2074,2568],{"dataTid":2072,"className":2073},"grU8\u002FiN1f8bgeG2W\u002F3+Sktw",[105],[93,2075,2077],{"transform":108,"dataTid":2076},"p\u002FraMdacpU4wMAz3KbNLcq8",[93,2078,2081,2100,2119,2138,2157,2176,2195,2210,2229,2244,2262,2281,2296,2311,2331,2346,2361,2376,2391,2406,2424,2439,2454,2469,2484,2499,2514,2529,2548,2563],{"dataTid":2079,"className":2080},"g\u002FraMdacpU4wMAz3KbNLcq8",[105],[93,2082,2085],{"transform":2083,"dataTid":2084},"translate(0.000,8.228)","pQ8c3N++zRuYuVmVghsYSmI",[93,2086,2089,2092],{"dataTid":2087,"className":2088,"fill":130,"transform":131},"gQ8c3N++zRuYuVmVghsYSmI",[129],[133,2090],{"x":135,"y":135,"href":2091},"#gl3Hi1RE",[93,2093,2094],{"transform":139},[141,2095,2097],{"x":135,"y":143,"width":2096,"height":145},"31.25",[147,2098,135],{"className":2099,"style":151},[150],[93,2101,2104],{"transform":2102,"dataTid":2103},"translate(5.500,8.228)","pymK4cOsYxgghlAviaG2xss",[93,2105,2108,2111],{"dataTid":2106,"className":2107,"fill":130,"transform":131},"gymK4cOsYxgghlAviaG2xss",[129],[133,2109],{"x":135,"y":135,"href":2110},"#gl3Hi1Q8",[93,2112,2113],{"transform":139},[141,2114,2116],{"x":135,"y":143,"width":2115,"height":145},"17.38",[147,2117,1967],{"className":2118,"style":151},[150],[93,2120,2123],{"transform":2121,"dataTid":2122},"translate(8.558,8.228)","pjxPYURMtyRcgim89k0Pu8I",[93,2124,2127,2130],{"dataTid":2125,"className":2126,"fill":130,"transform":131},"gjxPYURMtyRcgim89k0Pu8I",[129],[133,2128],{"x":135,"y":135,"href":2129},"#gl3Hi1RM",[93,2131,2132],{"transform":139},[141,2133,2134],{"x":135,"y":143,"width":2096,"height":145},[147,2135,2137],{"className":2136,"style":151},[150],"2",[93,2139,2142],{"transform":2140,"dataTid":2141},"translate(14.058,8.228)","pKHCR5FMmrqoPk3GJbswmKE",[93,2143,2146,2149],{"dataTid":2144,"className":2145,"fill":130,"transform":131},"gKHCR5FMmrqoPk3GJbswmKE",[129],[133,2147],{"x":135,"y":135,"href":2148},"#gl3Hi1RY",[93,2150,2151],{"transform":139},[141,2152,2153],{"x":135,"y":143,"width":2096,"height":145},[147,2154,2156],{"className":2155,"style":151},[150],"5",[93,2158,2161],{"transform":2159,"dataTid":2160},"translate(22.002,8.228)","pZRSO6c4p+nkXmDMBtidCsg",[93,2162,2165,2168],{"dataTid":2163,"className":2164,"fill":130,"transform":131},"gZRSO6c4p+nkXmDMBtidCsg",[129],[133,2166],{"x":135,"y":135,"href":2167},"#gl3Hi1Zg",[93,2169,2170],{"transform":139},[141,2171,2172],{"x":135,"y":143,"width":396,"height":145},[147,2173,2175],{"className":2174,"style":151},[150],"×",[93,2177,2180],{"transform":2178,"dataTid":2179},"translate(33.005,8.228)","psVZyKwl7TJ0WP15eV58nhs",[93,2181,2184,2187],{"dataTid":2182,"className":2183,"fill":130,"transform":131},"gsVZyKwl7TJ0WP15eV58nhs",[129],[133,2185],{"x":135,"y":135,"href":2186},"#gl3Hi1Qk",[93,2188,2189],{"transform":139},[141,2190,2192],{"x":135,"y":143,"width":2191,"height":145},"24.31",[147,2193,1227],{"className":2194,"style":151},[150],[93,2196,2198],{"transform":2197,"dataTid":2122},"translate(37.284,8.228)",[93,2199,2201,2203],{"dataTid":2125,"className":2200,"fill":130,"transform":131},[129],[133,2202],{"x":135,"y":135,"href":2129},[93,2204,2205],{"transform":139},[141,2206,2207],{"x":135,"y":143,"width":2096,"height":145},[147,2208,2137],{"className":2209,"style":151},[150],[93,2211,2214],{"transform":2212,"dataTid":2213},"translate(42.784,8.228)","pUAb\u002F9lCbaBg2WDdv8AUOhk",[93,2215,2218,2221],{"dataTid":2216,"className":2217,"fill":130,"transform":131},"gUAb\u002F9lCbaBg2WDdv8AUOhk",[129],[133,2219],{"x":135,"y":135,"href":2220},"#gl3Hi1Q0",[93,2222,2223],{"transform":139},[141,2224,2225],{"x":135,"y":143,"width":2115,"height":145},[147,2226,2228],{"className":2227,"style":151},[150],",",[93,2230,2232],{"transform":2231,"dataTid":2084},"translate(47.675,8.228)",[93,2233,2235,2237],{"dataTid":2087,"className":2234,"fill":130,"transform":131},[129],[133,2236],{"x":135,"y":135,"href":2091},[93,2238,2239],{"transform":139},[141,2240,2241],{"x":135,"y":143,"width":2096,"height":145},[147,2242,135],{"className":2243,"style":151},[150],[93,2245,2248],{"transform":2246,"dataTid":2247},"translate(53.175,8.228)","pXsuBVQM01DMq13gbLIiD0g",[93,2249,2252,2255],{"dataTid":2250,"className":2251,"fill":130,"transform":131},"gXsuBVQM01DMq13gbLIiD0g",[129],[133,2253],{"x":135,"y":135,"href":2254},"#gl3Hi1Qo",[93,2256,2257],{"transform":139},[141,2258,2259],{"x":135,"y":143,"width":2191,"height":145},[147,2260,1390],{"className":2261,"style":151},[150],[93,2263,2266],{"transform":2264,"dataTid":2265},"translate(59.899,8.228)","pzUvlOB5pueI9GCCk24jGCA",[93,2267,2270,2273],{"dataTid":2268,"className":2269,"fill":130,"transform":131},"gzUvlOB5pueI9GCCk24jGCA",[129],[133,2271],{"x":135,"y":135,"href":2272},"#gl3Hi1Qw",[93,2274,2275],{"transform":139},[141,2276,2277],{"x":135,"y":143,"width":396,"height":145},[147,2278,2280],{"className":2279,"style":151},[150],"+",[93,2282,2284],{"transform":2283,"dataTid":2084},"translate(70.901,8.228)",[93,2285,2287,2289],{"dataTid":2087,"className":2286,"fill":130,"transform":131},[129],[133,2288],{"x":135,"y":135,"href":2091},[93,2290,2291],{"transform":139},[141,2292,2293],{"x":135,"y":143,"width":2096,"height":145},[147,2294,135],{"className":2295,"style":151},[150],[93,2297,2299],{"transform":2298,"dataTid":2103},"translate(76.401,8.228)",[93,2300,2302,2304],{"dataTid":2106,"className":2301,"fill":130,"transform":131},[129],[133,2303],{"x":135,"y":135,"href":2110},[93,2305,2306],{"transform":139},[141,2307,2308],{"x":135,"y":143,"width":2115,"height":145},[147,2309,1967],{"className":2310,"style":151},[150],[93,2312,2315],{"transform":2313,"dataTid":2314},"translate(79.459,8.228)","pBwt+WiqGhhwMVTpkMMPvhk",[93,2316,2319,2322],{"dataTid":2317,"className":2318,"fill":130,"transform":131},"gBwt+WiqGhhwMVTpkMMPvhk",[129],[133,2320],{"x":135,"y":135,"href":2321},"#gl3Hi1Rg",[93,2323,2324],{"transform":139},[141,2325,2327],{"x":135,"y":143,"width":2326,"height":145},"32.06",[147,2328,2330],{"className":2329,"style":151},[150],"7",[93,2332,2334],{"transform":2333,"dataTid":2141},"translate(85.102,8.228)",[93,2335,2337,2339],{"dataTid":2144,"className":2336,"fill":130,"transform":131},[129],[133,2338],{"x":135,"y":135,"href":2148},[93,2340,2341],{"transform":139},[141,2342,2343],{"x":135,"y":143,"width":2096,"height":145},[147,2344,2156],{"className":2345,"style":151},[150],[93,2347,2349],{"transform":2348,"dataTid":2160},"translate(93.047,8.228)",[93,2350,2352,2354],{"dataTid":2163,"className":2351,"fill":130,"transform":131},[129],[133,2353],{"x":135,"y":135,"href":2167},[93,2355,2356],{"transform":139},[141,2357,2358],{"x":135,"y":143,"width":396,"height":145},[147,2359,2175],{"className":2360,"style":151},[150],[93,2362,2364],{"transform":2363,"dataTid":2179},"translate(104.049,8.228)",[93,2365,2367,2369],{"dataTid":2182,"className":2366,"fill":130,"transform":131},[129],[133,2368],{"x":135,"y":135,"href":2186},[93,2370,2371],{"transform":139},[141,2372,2373],{"x":135,"y":143,"width":2191,"height":145},[147,2374,1227],{"className":2375,"style":151},[150],[93,2377,2379],{"transform":2378,"dataTid":2084},"translate(108.328,8.228)",[93,2380,2382,2384],{"dataTid":2087,"className":2381,"fill":130,"transform":131},[129],[133,2383],{"x":135,"y":135,"href":2091},[93,2385,2386],{"transform":139},[141,2387,2388],{"x":135,"y":143,"width":2096,"height":145},[147,2389,135],{"className":2390,"style":151},[150],[93,2392,2394],{"transform":2393,"dataTid":2213},"translate(113.828,8.228)",[93,2395,2397,2399],{"dataTid":2216,"className":2396,"fill":130,"transform":131},[129],[133,2398],{"x":135,"y":135,"href":2220},[93,2400,2401],{"transform":139},[141,2402,2403],{"x":135,"y":143,"width":2115,"height":145},[147,2404,2228],{"className":2405,"style":151},[150],[93,2407,2410],{"transform":2408,"dataTid":2409},"translate(118.719,8.228)","pmcl2R9sRLT4\u002FZGwJ4fuqQM",[93,2411,2414,2417],{"dataTid":2412,"className":2413,"fill":130,"transform":131},"gmcl2R9sRLT4\u002FZGwJ4fuqQM",[129],[133,2415],{"x":135,"y":135,"href":2416},"#gl3Hi1RU",[93,2418,2419],{"transform":139},[141,2420,2421],{"x":135,"y":143,"width":2096,"height":145},[147,2422,99],{"className":2423,"style":151},[150],[93,2425,2427],{"transform":2426,"dataTid":2247},"translate(124.219,8.228)",[93,2428,2430,2432],{"dataTid":2250,"className":2429,"fill":130,"transform":131},[129],[133,2431],{"x":135,"y":135,"href":2254},[93,2433,2434],{"transform":139},[141,2435,2436],{"x":135,"y":143,"width":2191,"height":145},[147,2437,1390],{"className":2438,"style":151},[150],[93,2440,2442],{"transform":2441,"dataTid":384},"translate(131.554,8.228)",[93,2443,2445,2447],{"dataTid":387,"className":2444,"fill":130,"transform":131},[129],[133,2446],{"x":135,"y":135,"href":391},[93,2448,2449],{"transform":139},[141,2450,2451],{"x":135,"y":143,"width":396,"height":145},[147,2452,400],{"className":2453,"style":151},[150],[93,2455,2457],{"transform":2456,"dataTid":2179},"translate(143.167,8.228)",[93,2458,2460,2462],{"dataTid":2182,"className":2459,"fill":130,"transform":131},[129],[133,2461],{"x":135,"y":135,"href":2186},[93,2463,2464],{"transform":139},[141,2465,2466],{"x":135,"y":143,"width":2191,"height":145},[147,2467,1227],{"className":2468,"style":151},[150],[93,2470,2472],{"transform":2471,"dataTid":2084},"translate(147.446,8.228)",[93,2473,2475,2477],{"dataTid":2087,"className":2474,"fill":130,"transform":131},[129],[133,2476],{"x":135,"y":135,"href":2091},[93,2478,2479],{"transform":139},[141,2480,2481],{"x":135,"y":143,"width":2096,"height":145},[147,2482,135],{"className":2483,"style":151},[150],[93,2485,2487],{"transform":2486,"dataTid":2103},"translate(152.946,8.228)",[93,2488,2490,2492],{"dataTid":2106,"className":2489,"fill":130,"transform":131},[129],[133,2491],{"x":135,"y":135,"href":2110},[93,2493,2494],{"transform":139},[141,2495,2496],{"x":135,"y":143,"width":2115,"height":145},[147,2497,1967],{"className":2498,"style":151},[150],[93,2500,2502],{"transform":2501,"dataTid":2141},"translate(156.004,8.228)",[93,2503,2505,2507],{"dataTid":2144,"className":2504,"fill":130,"transform":131},[129],[133,2506],{"x":135,"y":135,"href":2148},[93,2508,2509],{"transform":139},[141,2510,2511],{"x":135,"y":143,"width":2096,"height":145},[147,2512,2156],{"className":2513,"style":151},[150],[93,2515,2517],{"transform":2516,"dataTid":2213},"translate(161.504,8.228)",[93,2518,2520,2522],{"dataTid":2216,"className":2519,"fill":130,"transform":131},[129],[133,2521],{"x":135,"y":135,"href":2220},[93,2523,2524],{"transform":139},[141,2525,2526],{"x":135,"y":143,"width":2115,"height":145},[147,2527,2228],{"className":2528,"style":151},[150],[93,2530,2533],{"transform":2531,"dataTid":2532},"translate(166.396,8.228)","pcyz+fVu\u002F6OA2tgmHhlMx1E",[93,2534,2537,2540],{"dataTid":2535,"className":2536,"fill":130,"transform":131},"gcyz+fVu\u002F6OA2tgmHhlMx1E",[129],[133,2538],{"x":135,"y":135,"href":2539},"#gl3Hi1RQ",[93,2541,2542],{"transform":139},[141,2543,2544],{"x":135,"y":143,"width":2096,"height":145},[147,2545,2547],{"className":2546,"style":151},[150],"3",[93,2549,2551],{"transform":2550,"dataTid":2247},"translate(171.896,8.228)",[93,2552,2554,2556],{"dataTid":2250,"className":2553,"fill":130,"transform":131},[129],[133,2555],{"x":135,"y":135,"href":2254},[93,2557,2558],{"transform":139},[141,2559,2560],{"x":135,"y":143,"width":2191,"height":145},[147,2561,1390],{"className":2562,"style":151},[150],[93,2564,2565],{"transform":108,"dataTid":155},[93,2566],{"dataTid":158,"className":2567,"dataHint":18},[160],[93,2569,2570],{"transform":108,"dataTid":155},[93,2571],{"dataTid":158,"className":2572,"dataHint":18},[160],[11,2574,2575],{},"These numbers are chosen to show the operation. They are not measurements from a model. The result contains information from both positions, with a larger contribution from the second.",[11,2577,2578],{},[2579,2580],"img",{"alt":2581,"src":2582},"The current token state is projected into a query, key and value. The query is compared with the keys through the current position. Softmax gives weights that combine the value vectors into one head's output.","\u002Fassets\u002Fblog\u002Fthe-hidden-memory-cost-of-llm-inference-kv-cache-explained\u002Fqkv-attention.svg",[11,2584,2585],{},"The model joins the outputs from its heads, then applies another learned transformation. More operations and layers follow. Only after the final layer does the model produce scores over its vocabulary. The engine uses those scores to choose the next token. The attention weights above describe how positions contribute to a head, rather than the probabilities of choosing output tokens.",[24,2587,2589],{"id":2588},"why-keys-and-values-stay-useful","Why keys and values stay useful",[11,2591,2592],{},"Causal attention prevents each position from reading future positions. If a sequence has five tokens and a sixth is added, the first five still depend only on the same earlier tokens and their own positions. With the same model and positional settings, their keys and values stay unchanged. Each layer can keep those vectors and reuse them.",[11,2594,2595],{},"At the sixth position, a layer computes a new query, key and value. It adds the new key and value to the stored ones. The new query is then compared with keys from all six positions, and the resulting weights combine their values. Every attention layer has its own cache because it works with a different representation of the sequence.",[11,2597,2598,2599,2604],{},"There is no need to keep the old queries for this step. They were used to compute the earlier positions. The new position needs its own query and the keys and values it can attend to. That is why the cache is called KV rather than QKV. The ",[18,2600,2603],{"href":2601,"rel":2602},"https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftransformers\u002Fcache_explanation",[985],"Transformers cache explanation"," describes this reuse at each layer.",[11,2606,2607],{},"The first pass is called prefill. It processes the prompt, builds the cache and produces the scores used to choose the first output token. Prompt positions can be processed in parallel, with a causal mask preventing each position from reading later ones. A serving engine may split a long prefill into smaller chunks.",[11,2609,2610],{},"Decode then processes the token just chosen. Its keys and values are added to the cache, and its final representation gives the scores used to choose the following token. This distinction matters. A generated token gets its own cached state when it is fed back into the model, not when it is first selected from the scores.",[11,2612,2613],{},[2579,2614],{"alt":2615,"src":2616},"During prefill the model writes keys and values for every prompt position. During decode each step appends one position and reads all stored ones.","\u002Fassets\u002Fblog\u002Fthe-hidden-memory-cost-of-llm-inference-kv-cache-explained\u002Fkv-cache-prefill-decode.svg",[11,2618,2619],{},"Caching saves repeated projections and the earlier layer calculations needed to produce them. It does not remove attention over the retained context. With full attention, a longer cache means more keys and values to read for each new position.",[24,2621,2623],{"id":2622},"counting-memory-one-vector-at-a-time","Counting memory one vector at a time",[11,2625,2626,2627,2678,2679,2730,2731,2782,2783,2880],{},"Start with one cached token in one layer and one KV head. If each vector contains ",[56,2628,2630,2632,2638,2641,2643],{"style":58,"className":2629,"viewBox":1622,"width":1623,"height":63,"dataWidth":1624,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],[70,2631,73],{"type":72},[75,2633,2635],{"className":2634},[78],[80,2636],{"id":1056,"className":2637,"d":1058},[84],[75,2639],{"className":2640},[89],[70,2642],{"type":72},[93,2644,2646],{"className":2645,"transform":97,"dataTid":1641,"dataPageWidth":1642,"dataPageHeight":100},[96],[93,2647,2649],{"dataTid":1645,"className":2648},[105],[93,2650,2651],{"transform":108,"dataTid":1649},[93,2652,2654,2673],{"dataTid":1652,"className":2653},[105],[93,2655,2656],{"transform":108,"dataTid":1656},[93,2657,2659],{"dataTid":1659,"className":2658},[105],[93,2660,2661],{"transform":123,"dataTid":1344},[93,2662,2664,2666],{"dataTid":1347,"className":2663,"fill":130,"transform":131},[129],[133,2665],{"x":135,"y":135,"href":1351},[93,2667,2668],{"transform":139},[141,2669,2670],{"x":135,"y":143,"width":1356,"height":145},[147,2671,1360],{"className":2672,"style":151},[150],[93,2674,2675],{"transform":108,"dataTid":155},[93,2676],{"dataTid":158,"className":2677,"dataHint":18},[160]," numbers, its key takes ",[56,2680,2682,2684,2690,2693,2695],{"style":58,"className":2681,"viewBox":1622,"width":1623,"height":63,"dataWidth":1624,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],[70,2683,73],{"type":72},[75,2685,2687],{"className":2686},[78],[80,2688],{"id":1056,"className":2689,"d":1058},[84],[75,2691],{"className":2692},[89],[70,2694],{"type":72},[93,2696,2698],{"className":2697,"transform":97,"dataTid":1641,"dataPageWidth":1642,"dataPageHeight":100},[96],[93,2699,2701],{"dataTid":1645,"className":2700},[105],[93,2702,2703],{"transform":108,"dataTid":1649},[93,2704,2706,2725],{"dataTid":1652,"className":2705},[105],[93,2707,2708],{"transform":108,"dataTid":1656},[93,2709,2711],{"dataTid":1659,"className":2710},[105],[93,2712,2713],{"transform":123,"dataTid":1344},[93,2714,2716,2718],{"dataTid":1347,"className":2715,"fill":130,"transform":131},[129],[133,2717],{"x":135,"y":135,"href":1351},[93,2719,2720],{"transform":139},[141,2721,2722],{"x":135,"y":143,"width":1356,"height":145},[147,2723,1360],{"className":2724,"style":151},[150],[93,2726,2727],{"transform":108,"dataTid":155},[93,2728],{"dataTid":158,"className":2729,"dataHint":18},[160]," numbers and its value takes another ",[56,2732,2734,2736,2742,2745,2747],{"style":58,"className":2733,"viewBox":1622,"width":1623,"height":63,"dataWidth":1624,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],[70,2735,73],{"type":72},[75,2737,2739],{"className":2738},[78],[80,2740],{"id":1056,"className":2741,"d":1058},[84],[75,2743],{"className":2744},[89],[70,2746],{"type":72},[93,2748,2750],{"className":2749,"transform":97,"dataTid":1641,"dataPageWidth":1642,"dataPageHeight":100},[96],[93,2751,2753],{"dataTid":1645,"className":2752},[105],[93,2754,2755],{"transform":108,"dataTid":1649},[93,2756,2758,2777],{"dataTid":1652,"className":2757},[105],[93,2759,2760],{"transform":108,"dataTid":1656},[93,2761,2763],{"dataTid":1659,"className":2762},[105],[93,2764,2765],{"transform":123,"dataTid":1344},[93,2766,2768,2770],{"dataTid":1347,"className":2767,"fill":130,"transform":131},[129],[133,2769],{"x":135,"y":135,"href":1351},[93,2771,2772],{"transform":139},[141,2773,2774],{"x":135,"y":143,"width":1356,"height":145},[147,2775,1360],{"className":2776,"style":151},[150],[93,2778,2779],{"transform":108,"dataTid":155},[93,2780],{"dataTid":158,"className":2781,"dataHint":18},[160],". Together they take ",[56,2784,2789,2791,2803,2806,2808],{"style":58,"className":2785,"viewBox":2786,"width":2787,"height":63,"dataWidth":2788,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],"0 0 25.000 19.000","2.272727272727273em","25.000",[70,2790,73],{"type":72},[75,2792,2794,2797,2800],{"className":2793},[78],[80,2795],{"id":2022,"className":2796,"d":2024},[84],[80,2798],{"id":2050,"className":2799,"d":2052},[84],[80,2801],{"id":1056,"className":2802,"d":1058},[84],[75,2804],{"className":2805},[89],[70,2807],{"type":72},[93,2809,2813],{"className":2810,"transform":97,"dataTid":2811,"dataPageWidth":2812,"dataPageHeight":100},[96],"pcFj8BOXJBCkCNOY1FGYP88","25",[93,2814,2817],{"dataTid":2815,"className":2816},"gcFj8BOXJBCkCNOY1FGYP88",[105],[93,2818,2820],{"transform":108,"dataTid":2819},"pziBe69VtdkkzXzwmkMO6eg",[93,2821,2824,2875],{"dataTid":2822,"className":2823},"gziBe69VtdkkzXzwmkMO6eg",[105],[93,2825,2827],{"transform":108,"dataTid":2826},"p8qfCD\u002FB5Q98FONtjPTHE+M",[93,2828,2831,2845,2860],{"dataTid":2829,"className":2830},"g8qfCD\u002FB5Q98FONtjPTHE+M",[105],[93,2832,2833],{"transform":123,"dataTid":2122},[93,2834,2836,2838],{"dataTid":2125,"className":2835,"fill":130,"transform":131},[129],[133,2837],{"x":135,"y":135,"href":2129},[93,2839,2840],{"transform":139},[141,2841,2842],{"x":135,"y":143,"width":2096,"height":145},[147,2843,2137],{"className":2844,"style":151},[150],[93,2846,2848],{"transform":2847,"dataTid":2160},"translate(7.944,13.013)",[93,2849,2851,2853],{"dataTid":2163,"className":2850,"fill":130,"transform":131},[129],[133,2852],{"x":135,"y":135,"href":2167},[93,2854,2855],{"transform":139},[141,2856,2857],{"x":135,"y":143,"width":396,"height":145},[147,2858,2175],{"className":2859,"style":151},[150],[93,2861,2863],{"transform":2862,"dataTid":1344},"translate(18.947,13.013)",[93,2864,2866,2868],{"dataTid":1347,"className":2865,"fill":130,"transform":131},[129],[133,2867],{"x":135,"y":135,"href":1351},[93,2869,2870],{"transform":139},[141,2871,2872],{"x":135,"y":143,"width":1356,"height":145},[147,2873,1360],{"className":2874,"style":151},[150],[93,2876,2877],{"transform":108,"dataTid":155},[93,2878],{"dataTid":158,"className":2879,"dataHint":18},[160]," numbers. The factor of two comes from storing both K and V.",[11,2882,2883,2884,2986,2987,3138,3139,3207,3208,3392],{},"If that layer has ",[56,2885,2890,2892,2910,2913,2915],{"style":58,"className":2886,"viewBox":2887,"width":2888,"height":63,"dataWidth":2889,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],"0 0 22.000 19.000","2em","22.000",[70,2891,73],{"type":72},[75,2893,2895,2900,2905],{"className":2894},[78],[80,2896],{"id":2897,"className":2898,"d":2899},"gl3Hi1Sw",[84],"M 550 616 C 550 607 545 597 534 586 L 231 297 L 231 602 C 231 620 235 631 242 636 C 249 641 270 644 305 644 L 333 644 L 333 683 C 307 681 257 680 183 680 C 109 680 59 681 33 683 L 33 644 L 61 644 C 96 644 117 641 124 636 C 131 631 135 620 135 602 L 135 81 C 135 63 131 51 124 46 C 117 41 96 39 61 39 L 33 39 L 33 0 C 59 2 109 3 182 3 C 257 3 307 2 333 0 L 333 39 L 305 39 C 270 39 249 41 242 46 C 235 51 231 63 231 81 L 231 252 L 341 357 L 517 96 C 526 82 531 71 531 64 C 531 47 512 39 473 39 L 473 0 C 516 3 564 4 618 3 L 736 0 L 736 39 C 671 39 659 45 631 84 L 405 418 L 581 586 C 620 625 667 644 722 644 L 722 683 C 702 681 673 680 634 680 C 573 680 530 681 505 683 L 505 644 C 529 644 550 636 550 616 Z ",[80,2901],{"id":2902,"className":2903,"d":2904},"gl3Hi1Tc",[84],"M 622 581 C 638 622 674 643 730 644 L 730 683 C 687 680 654 679 630 680 L 513 683 L 513 645 C 562 644 586 628 586 599 C 586 595 584 589 581 581 L 404 114 L 218 603 C 215 613 213 619 213 620 C 213 636 240 644 294 644 L 294 683 C 252 680 204 679 149 680 L 19 683 L 19 644 C 56 644 80 642 91 639 C 102 636 110 626 116 609 L 346 2 C 352 -14 361 -22 374 -22 C 387 -22 396 -15 401 -1 Z ",[80,2906],{"id":2907,"className":2908,"d":2909},"gl3Hi1fUK",[84],"M 881 668 C 881 678 875 683 863 683 L 736 680 L 609 683 C 593 683 586 674 586 659 C 586 652 589 647 594 646 C 604 645 612 644 617 644 C 648 643 665 641 670 640 C 675 639 678 636 678 631 C 677 628 676 622 674 613 L 615 375 L 321 375 L 379 602 C 384 623 393 636 406 641 C 413 643 430 644 458 644 C 485 644 496 644 496 668 C 496 678 490 683 478 683 L 351 680 L 223 683 C 207 683 200 674 200 659 C 200 652 203 647 209 646 C 219 645 227 644 232 644 C 263 643 281 641 286 640 C 291 639 293 636 293 631 C 293 629 292 623 289 613 L 156 82 C 151 60 141 47 126 42 C 119 40 101 39 70 39 C 48 39 39 37 39 15 C 39 5 45 0 57 0 L 183 3 L 246 2 C 257 2 299 0 310 0 C 326 0 334 8 334 24 C 334 34 323 39 302 39 C 262 39 242 43 242 52 C 242 52 243 55 245 68 L 312 336 L 605 336 L 538 68 C 535 53 523 43 503 40 C 498 39 481 39 452 39 C 433 39 424 31 424 15 C 424 5 430 0 442 0 L 568 3 L 631 2 C 642 2 683 0 696 0 C 712 0 720 8 720 24 C 720 34 709 39 688 39 C 647 39 627 43 627 52 C 627 52 628 55 630 68 L 764 602 C 769 623 778 636 791 641 C 797 643 814 644 843 644 C 869 644 881 644 881 668 Z ",[75,2911],{"className":2912},[89],[70,2914],{"type":72},[93,2916,2920],{"className":2917,"transform":97,"dataTid":2918,"dataPageWidth":2919,"dataPageHeight":100},[96],"pLh5YJEmSXGcJsqChbk0AZA","22",[93,2921,2924],{"dataTid":2922,"className":2923},"gLh5YJEmSXGcJsqChbk0AZA",[105],[93,2925,2927],{"transform":108,"dataTid":2926},"pzys74zf5xX4kuCSC1WWSPM",[93,2928,2931,2981],{"dataTid":2929,"className":2930},"gzys74zf5xX4kuCSC1WWSPM",[105],[93,2932,2934],{"transform":108,"dataTid":2933},"pQfY5WMb6mGMgEHrl9PvhqU",[93,2935,2938,2957],{"dataTid":2936,"className":2937},"gQfY5WMb6mGMgEHrl9PvhqU",[105],[93,2939,2941],{"transform":123,"dataTid":2940},"pzflOl9Rc+LkJgayB5QZbX8",[93,2942,2945,2948],{"dataTid":2943,"className":2944,"fill":130,"transform":131},"gzflOl9Rc+LkJgayB5QZbX8",[129],[133,2946],{"x":135,"y":135,"href":2947},"#gl3Hi1fUK",[93,2949,2950],{"transform":139},[141,2951,2953],{"x":135,"y":143,"width":2952,"height":145},"56.81",[147,2954,2956],{"className":2955,"style":151},[150],"𝐻",[93,2958,2961],{"transform":2959,"dataTid":2960},"translate(8.921,15.730)","peKm4Q78clr43UIBMNjxQR4",[93,2962,2965,2968,2972],{"dataTid":2963,"className":2964,"fill":130,"transform":238},"geKm4Q78clr43UIBMNjxQR4",[129],[133,2966],{"x":135,"y":135,"href":2967},"#gl3Hi1Sw",[133,2969],{"x":2970,"y":135,"href":2971},"778","#gl3Hi1Tc",[93,2973,2974],{"transform":139},[141,2975,2977],{"x":135,"y":143,"width":2976,"height":145},"95.50",[147,2978,2980],{"className":2979,"style":151},[150],"KV",[93,2982,2983],{"transform":108,"dataTid":155},[93,2984],{"dataTid":158,"className":2985,"dataHint":18},[160]," KV heads, it needs that pair of vectors for every head. The count becomes ",[56,2988,2993,2995,3016,3019,3021],{"style":58,"className":2989,"viewBox":2990,"width":2991,"height":63,"dataWidth":2992,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],"0 0 60.000 19.000","5.454545454545454em","60.000",[70,2994,73],{"type":72},[75,2996,2998,3001,3004,3007,3010,3013],{"className":2997},[78],[80,2999],{"id":2022,"className":3000,"d":2024},[84],[80,3002],{"id":2897,"className":3003,"d":2899},[84],[80,3005],{"id":2902,"className":3006,"d":2904},[84],[80,3008],{"id":2050,"className":3009,"d":2052},[84],[80,3011],{"id":2907,"className":3012,"d":2909},[84],[80,3014],{"id":1056,"className":3015,"d":1058},[84],[75,3017],{"className":3018},[89],[70,3020],{"type":72},[93,3022,3026],{"className":3023,"transform":97,"dataTid":3024,"dataPageWidth":3025,"dataPageHeight":100},[96],"pcx\u002Fckv3jqRkLiK\u002FccRTQN4","60",[93,3027,3030],{"dataTid":3028,"className":3029},"gcx\u002Fckv3jqRkLiK\u002FccRTQN4",[105],[93,3031,3033],{"transform":108,"dataTid":3032},"pwKaoGXYPvjcw43bJOYRqqA",[93,3034,3037,3133],{"dataTid":3035,"className":3036},"gwKaoGXYPvjcw43bJOYRqqA",[105],[93,3038,3040],{"transform":108,"dataTid":3039},"pbv7oghNy5sklPiTckRpqfc",[93,3041,3044,3058,3072,3086,3103,3118],{"dataTid":3042,"className":3043},"gbv7oghNy5sklPiTckRpqfc",[105],[93,3045,3046],{"transform":123,"dataTid":2122},[93,3047,3049,3051],{"dataTid":2125,"className":3048,"fill":130,"transform":131},[129],[133,3050],{"x":135,"y":135,"href":2129},[93,3052,3053],{"transform":139},[141,3054,3055],{"x":135,"y":143,"width":2096,"height":145},[147,3056,2137],{"className":3057,"style":151},[150],[93,3059,3060],{"transform":2847,"dataTid":2160},[93,3061,3063,3065],{"dataTid":2163,"className":3062,"fill":130,"transform":131},[129],[133,3064],{"x":135,"y":135,"href":2167},[93,3066,3067],{"transform":139},[141,3068,3069],{"x":135,"y":143,"width":396,"height":145},[147,3070,2175],{"className":3071,"style":151},[150],[93,3073,3074],{"transform":2862,"dataTid":2940},[93,3075,3077,3079],{"dataTid":2943,"className":3076,"fill":130,"transform":131},[129],[133,3078],{"x":135,"y":135,"href":2947},[93,3080,3081],{"transform":139},[141,3082,3083],{"x":135,"y":143,"width":2952,"height":145},[147,3084,2956],{"className":3085,"style":151},[150],[93,3087,3089],{"transform":3088,"dataTid":2960},"translate(27.868,15.730)",[93,3090,3092,3094,3096],{"dataTid":2963,"className":3091,"fill":130,"transform":238},[129],[133,3093],{"x":135,"y":135,"href":2967},[133,3095],{"x":2970,"y":135,"href":2971},[93,3097,3098],{"transform":139},[141,3099,3100],{"x":135,"y":143,"width":2976,"height":145},[147,3101,2980],{"className":3102,"style":151},[150],[93,3104,3106],{"transform":3105,"dataTid":2160},"translate(42.694,13.013)",[93,3107,3109,3111],{"dataTid":2163,"className":3108,"fill":130,"transform":131},[129],[133,3110],{"x":135,"y":135,"href":2167},[93,3112,3113],{"transform":139},[141,3114,3115],{"x":135,"y":143,"width":396,"height":145},[147,3116,2175],{"className":3117,"style":151},[150],[93,3119,3121],{"transform":3120,"dataTid":1344},"translate(53.696,13.013)",[93,3122,3124,3126],{"dataTid":1347,"className":3123,"fill":130,"transform":131},[129],[133,3125],{"x":135,"y":135,"href":1351},[93,3127,3128],{"transform":139},[141,3129,3130],{"x":135,"y":143,"width":1356,"height":145},[147,3131,1360],{"className":3132,"style":151},[150],[93,3134,3135],{"transform":108,"dataTid":155},[93,3136],{"dataTid":158,"className":3137,"dataHint":18},[160],". With ",[56,3140,3145,3147,3155,3158,3160],{"style":58,"className":3141,"viewBox":3142,"width":3143,"height":63,"dataWidth":3144,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],"0 0 8.000 19.000","0.7272727272727273em","8.000",[70,3146,73],{"type":72},[75,3148,3150],{"className":3149},[78],[80,3151],{"id":3152,"className":3153,"d":3154},"gl3Hi1fkK",[84],"M 520 667 C 520 678 513 683 500 683 L 354 680 L 223 683 C 208 684 200 676 200 659 C 200 652 203 647 209 646 C 218 645 226 644 231 644 C 262 643 279 641 284 640 C 289 639 292 636 292 631 C 292 629 291 623 288 613 L 156 82 C 150 60 140 47 125 42 C 119 40 101 39 70 39 C 49 39 39 31 39 15 C 39 5 49 0 70 0 L 527 0 C 552 0 554 0 561 19 L 639 233 C 642 240 643 245 643 248 C 643 258 638 263 627 263 C 617 263 612 254 607 240 C 588 191 570 154 555 131 C 518 75 455 39 363 39 L 270 39 C 259 39 252 39 249 40 C 242 40 239 42 239 45 C 239 47 241 54 244 67 L 377 601 C 383 624 396 637 415 642 C 421 643 442 644 478 644 C 506 644 520 644 520 667 Z ",[75,3156],{"className":3157},[89],[70,3159],{"type":72},[93,3161,3165],{"className":3162,"transform":97,"dataTid":3163,"dataPageWidth":3164,"dataPageHeight":100},[96],"py2zaZ4fbVTkPbPpJKgovpc","8",[93,3166,3169],{"dataTid":3167,"className":3168},"gy2zaZ4fbVTkPbPpJKgovpc",[105],[93,3170,3172],{"transform":108,"dataTid":3171},"p1qmMDR0xUu8wMN1qBFW\u002F5Y",[93,3173,3176,3202],{"dataTid":3174,"className":3175},"g1qmMDR0xUu8wMN1qBFW\u002F5Y",[105],[93,3177,3179],{"transform":108,"dataTid":3178},"p6NIvD\u002FGbf5UyhciRDumE98",[93,3180,3183],{"dataTid":3181,"className":3182},"g6NIvD\u002FGbf5UyhciRDumE98",[105],[93,3184,3186],{"transform":123,"dataTid":3185},"pcKTCIYgQAtcg\u002FjmTeQxEpI",[93,3187,3190,3193],{"dataTid":3188,"className":3189,"fill":130,"transform":131},"gcKTCIYgQAtcg\u002FjmTeQxEpI",[129],[133,3191],{"x":135,"y":135,"href":3192},"#gl3Hi1fkK",[93,3194,3195],{"transform":139},[141,3196,3198],{"x":135,"y":143,"width":3197,"height":145},"42.56",[147,3199,3201],{"className":3200,"style":151},[150],"𝐿",[93,3203,3204],{"transform":108,"dataTid":155},[93,3205],{"dataTid":158,"className":3206,"dataHint":18},[160]," layers that all have the same shape and keep separate caches, it becomes ",[56,3209,3214,3216,3240,3243,3245],{"style":58,"className":3210,"viewBox":3211,"width":3212,"height":63,"dataWidth":3213,"dataHeight":65,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],"0 0 81.000 19.000","7.363636363636363em","81.000",[70,3215,73],{"type":72},[75,3217,3219,3222,3225,3228,3231,3234,3237],{"className":3218},[78],[80,3220],{"id":2022,"className":3221,"d":2024},[84],[80,3223],{"id":2897,"className":3224,"d":2899},[84],[80,3226],{"id":2902,"className":3227,"d":2904},[84],[80,3229],{"id":2050,"className":3230,"d":2052},[84],[80,3232],{"id":2907,"className":3233,"d":2909},[84],[80,3235],{"id":3152,"className":3236,"d":3154},[84],[80,3238],{"id":1056,"className":3239,"d":1058},[84],[75,3241],{"className":3242},[89],[70,3244],{"type":72},[93,3246,3250],{"className":3247,"transform":97,"dataTid":3248,"dataPageWidth":3249,"dataPageHeight":100},[96],"pfyk\u002F5yjyVcAMk5y8FljEoc","81",[93,3251,3254],{"dataTid":3252,"className":3253},"gfyk\u002F5yjyVcAMk5y8FljEoc",[105],[93,3255,3257],{"transform":108,"dataTid":3256},"pcEgqfEAxzfA5eXKeBr6Mqk",[93,3258,3261,3387],{"dataTid":3259,"className":3260},"gcEgqfEAxzfA5eXKeBr6Mqk",[105],[93,3262,3264],{"transform":108,"dataTid":3263},"paOHyO1Nk428GXVlkgKhw7Q",[93,3265,3268,3282,3296,3310,3325,3340,3357,3372],{"dataTid":3266,"className":3267},"gaOHyO1Nk428GXVlkgKhw7Q",[105],[93,3269,3270],{"transform":123,"dataTid":2122},[93,3271,3273,3275],{"dataTid":2125,"className":3272,"fill":130,"transform":131},[129],[133,3274],{"x":135,"y":135,"href":2129},[93,3276,3277],{"transform":139},[141,3278,3279],{"x":135,"y":143,"width":2096,"height":145},[147,3280,2137],{"className":3281,"style":151},[150],[93,3283,3284],{"transform":2847,"dataTid":2160},[93,3285,3287,3289],{"dataTid":2163,"className":3286,"fill":130,"transform":131},[129],[133,3288],{"x":135,"y":135,"href":2167},[93,3290,3291],{"transform":139},[141,3292,3293],{"x":135,"y":143,"width":396,"height":145},[147,3294,2175],{"className":3295,"style":151},[150],[93,3297,3298],{"transform":2862,"dataTid":3185},[93,3299,3301,3303],{"dataTid":3188,"className":3300,"fill":130,"transform":131},[129],[133,3302],{"x":135,"y":135,"href":3192},[93,3304,3305],{"transform":139},[141,3306,3307],{"x":135,"y":143,"width":3197,"height":145},[147,3308,3201],{"className":3309,"style":151},[150],[93,3311,3313],{"transform":3312,"dataTid":2160},"translate(28.882,13.013)",[93,3314,3316,3318],{"dataTid":2163,"className":3315,"fill":130,"transform":131},[129],[133,3317],{"x":135,"y":135,"href":2167},[93,3319,3320],{"transform":139},[141,3321,3322],{"x":135,"y":143,"width":396,"height":145},[147,3323,2175],{"className":3324,"style":151},[150],[93,3326,3328],{"transform":3327,"dataTid":2940},"translate(39.885,13.013)",[93,3329,3331,3333],{"dataTid":2943,"className":3330,"fill":130,"transform":131},[129],[133,3332],{"x":135,"y":135,"href":2947},[93,3334,3335],{"transform":139},[141,3336,3337],{"x":135,"y":143,"width":2952,"height":145},[147,3338,2956],{"className":3339,"style":151},[150],[93,3341,3343],{"transform":3342,"dataTid":2960},"translate(48.806,15.730)",[93,3344,3346,3348,3350],{"dataTid":2963,"className":3345,"fill":130,"transform":238},[129],[133,3347],{"x":135,"y":135,"href":2967},[133,3349],{"x":2970,"y":135,"href":2971},[93,3351,3352],{"transform":139},[141,3353,3354],{"x":135,"y":143,"width":2976,"height":145},[147,3355,2980],{"className":3356,"style":151},[150],[93,3358,3360],{"transform":3359,"dataTid":2160},"translate(63.632,13.013)",[93,3361,3363,3365],{"dataTid":2163,"className":3362,"fill":130,"transform":131},[129],[133,3364],{"x":135,"y":135,"href":2167},[93,3366,3367],{"transform":139},[141,3368,3369],{"x":135,"y":143,"width":396,"height":145},[147,3370,2175],{"className":3371,"style":151},[150],[93,3373,3375],{"transform":3374,"dataTid":1344},"translate(74.634,13.013)",[93,3376,3378,3380],{"dataTid":1347,"className":3377,"fill":130,"transform":131},[129],[133,3379],{"x":135,"y":135,"href":1351},[93,3381,3382],{"transform":139},[141,3383,3384],{"x":135,"y":143,"width":1356,"height":145},[147,3385,1360],{"className":3386,"style":151},[150],[93,3388,3389],{"transform":108,"dataTid":155},[93,3390],{"dataTid":158,"className":3391,"dataHint":18},[160]," numbers per token.",[11,3394,3395],{},"Next, multiply by the number of cached positions. This includes the prompt and the generated tokens already processed by the model. Finally, multiply by the bytes used for each number. A 16-bit cache uses 2 bytes per element. Its precision is a separate choice from the precision used for the model weights.",[11,3397,3398],{},"For several independent sequences of the same length, multiply by batch size, the number of sequences. Combining these factors gives",[56,3400,3407,3409,3507,3510,3512],{"style":258,"className":3401,"viewBox":3402,"width":3403,"height":3404,"dataWidth":3405,"dataHeight":3406,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],"0 0 207.000 42.000","18.818181818181817em","3.8181818181818183em","207.000","42.000",[70,3408,73],{"type":72},[75,3410,3412,3415,3418,3421,3424,3427,3432,3437,3442,3447,3452,3457,3462,3467,3470,3475,3478,3483,3488,3491,3494,3499,3504],{"className":3411},[78],[80,3413],{"id":2022,"className":3414,"d":2024},[84],[80,3416],{"id":272,"className":3417,"d":274},[84],[80,3419],{"id":2897,"className":3420,"d":2899},[84],[80,3422],{"id":2902,"className":3423,"d":2904},[84],[80,3425],{"id":1006,"className":3426,"d":1008},[84],[80,3428],{"id":3429,"className":3430,"d":3431},"gl3Hi1UM",[84],"M 298 -11 C 360 -11 413 11 456 56 C 499 101 521 154 521 217 C 521 278 501 332 460 377 C 419 422 369 445 308 445 C 259 445 215 426 178 387 L 178 694 L 28 683 L 28 645 C 64 645 86 642 94 636 C 102 630 105 616 105 593 L 105 0 L 136 0 L 169 58 C 201 12 244 -11 298 -11 Z M 294 19 C 253 19 219 38 194 77 C 185 90 181 104 181 117 L 181 320 C 181 333 184 344 191 354 C 220 395 257 415 304 415 C 345 415 378 395 404 354 C 422 324 431 279 431 218 C 431 110 393 19 294 19 Z ",[80,3433],{"id":3434,"className":3435,"d":3436},"gl3Hi1UQ",[84],"M 251 416 C 293 416 325 406 347 386 C 323 383 306 361 306 338 C 306 305 322 289 355 289 C 388 289 404 306 404 339 C 404 410 327 448 250 448 C 188 448 137 426 96 380 C 55 334 34 279 34 216 C 34 155 55 101 96 56 C 137 11 187 -11 248 -11 C 298 -11 337 3 365 32 C 388 55 403 78 411 102 C 414 111 415 117 415 121 C 415 130 409 134 398 134 C 390 134 385 130 382 122 C 361 55 320 21 257 21 C 228 21 200 34 172 61 C 139 92 123 144 123 218 C 123 318 161 416 251 416 Z ",[80,3438],{"id":3439,"className":3440,"d":3441},"gl3Hi1UU",[84],"M 374 -11 L 527 0 L 527 38 C 491 38 470 40 462 46 C 454 52 450 67 450 90 L 450 694 L 301 683 L 301 645 C 336 645 358 642 366 636 C 374 630 377 616 377 593 L 377 390 C 345 427 305 445 257 445 C 195 445 142 423 99 378 C 56 333 34 279 34 216 C 34 155 54 102 95 57 C 136 12 186 -11 247 -11 C 298 -11 341 8 374 47 Z M 261 415 C 304 415 339 396 364 358 C 371 347 374 336 374 323 L 374 121 C 374 108 371 97 364 86 C 336 41 298 19 251 19 C 210 19 177 39 151 80 C 133 110 124 155 124 215 C 124 324 161 415 261 415 Z ",[80,3443],{"id":3444,"className":3445,"d":3446},"gl3Hi1UY",[84],"M 415 251 C 415 366 348 448 236 448 C 176 448 126 425 85 378 C 47 334 28 281 28 220 C 28 157 49 104 91 58 C 133 12 185 -11 247 -11 C 304 -11 348 9 378 48 C 403 81 415 105 415 121 C 415 131 409 136 398 136 C 390 136 385 131 382 120 C 360 54 317 21 253 21 C 208 21 173 42 146 85 C 127 116 118 163 117 227 L 387 227 C 409 227 415 229 415 251 Z M 326 349 C 337 316 342 284 342 255 L 118 255 C 121 321 138 366 168 389 C 193 409 215 419 236 419 C 281 419 311 396 326 349 Z ",[80,3448],{"id":3449,"className":3450,"d":3451},"gl3Hi1Uk",[84],"M 314 415 C 359 415 382 380 382 309 L 382 79 C 382 60 379 48 372 44 C 365 40 343 39 306 39 L 306 0 L 421 3 L 535 0 L 535 39 C 503 39 482 40 472 43 C 462 46 458 53 458 64 L 458 253 C 458 301 457 334 454 353 C 444 414 399 445 320 445 C 259 445 213 417 182 361 L 182 694 L 32 683 L 32 645 C 68 645 90 642 98 636 C 106 630 109 616 109 593 L 109 79 C 109 60 105 48 98 44 C 91 40 69 39 32 39 L 32 0 L 147 3 L 261 0 L 261 39 C 224 39 203 40 196 44 C 189 48 185 60 185 79 L 185 261 C 185 342 236 415 314 415 Z ",[80,3453],{"id":3454,"className":3455,"d":3456},"gl3Hi1Uo",[84],"M 194 601 C 194 631 169 657 139 657 C 109 657 83 631 83 601 C 83 571 108 544 138 544 C 169 544 194 570 194 601 Z M 143 3 L 247 0 L 247 39 C 214 39 194 41 188 45 C 182 49 180 60 180 78 L 180 445 L 37 433 L 37 395 C 70 395 91 392 98 387 C 105 382 108 368 108 345 L 108 79 C 108 60 105 48 98 44 C 91 40 69 39 33 39 L 33 0 Z ",[80,3458],{"id":3459,"className":3460,"d":3461},"gl3Hi1Uw",[84],"M 422 3 C 442 4 472 3 511 0 L 511 39 C 487 39 469 42 457 47 C 445 52 431 66 415 89 L 290 266 C 287 275 285 278 285 275 C 285 276 295 285 316 303 C 351 334 377 354 394 365 C 425 384 456 394 488 396 L 488 434 C 467 432 443 431 416 431 C 370 431 332 432 301 434 L 301 396 C 320 395 330 388 330 375 C 330 363 323 351 310 339 L 178 224 L 178 694 L 28 683 L 28 645 C 64 645 86 642 94 636 C 102 630 105 616 105 593 L 105 79 C 105 60 101 48 94 44 C 87 40 65 39 28 39 L 28 0 L 139 3 L 251 0 L 251 39 C 214 39 193 40 186 44 C 179 48 175 60 175 79 L 175 179 L 233 230 L 249 206 C 314 116 347 66 347 57 C 347 45 335 39 311 39 L 311 0 Z ",[80,3463],{"id":3464,"className":3465,"d":3466},"gl3Hi1U0",[84],"M 144 3 L 255 0 L 255 39 C 219 39 197 40 190 44 C 183 48 180 60 180 79 L 180 694 L 33 683 L 33 645 C 68 645 90 642 97 636 C 104 630 108 616 108 593 L 108 79 C 108 60 105 48 98 44 C 91 40 69 39 33 39 L 33 0 Z ",[80,3468],{"id":1016,"className":3469,"d":1018},[84],[80,3471],{"id":3472,"className":3473,"d":3474},"gl3Hi1U8",[84],"M 314 413 C 359 413 382 378 382 307 L 382 79 C 382 60 379 48 372 44 C 365 40 343 38 306 38 L 306 0 L 421 3 L 535 0 L 535 38 C 503 38 482 39 472 42 C 462 45 458 52 458 64 L 458 251 C 458 298 457 331 454 350 C 444 411 399 442 320 442 C 257 442 210 412 179 352 L 179 442 L 32 431 L 32 393 C 68 393 90 390 98 384 C 106 378 109 365 109 342 L 109 79 C 109 60 105 48 98 44 C 91 40 69 38 32 38 L 32 0 L 147 3 L 261 0 L 261 38 C 224 38 203 40 196 44 C 189 48 185 60 185 79 L 185 259 C 185 340 236 413 314 413 Z ",[80,3476],{"id":1021,"className":3477,"d":1023},[84],[80,3479],{"id":3480,"className":3481,"d":3482},"gl3Hi1VE",[84],"M 298 -13 C 360 -13 413 9 456 54 C 499 99 521 152 521 215 C 521 276 501 328 462 372 C 421 419 371 442 311 442 C 260 442 216 422 178 383 L 178 442 L 28 431 L 28 393 C 65 393 87 391 94 386 C 101 381 105 368 105 347 L 105 -115 C 105 -134 101 -146 94 -150 C 87 -154 65 -155 28 -155 L 28 -194 L 142 -191 L 257 -194 L 257 -155 C 220 -155 199 -154 192 -150 C 185 -146 181 -134 181 -115 L 181 44 C 203 13 246 -13 298 -13 Z M 304 409 C 345 409 377 386 402 340 C 421 301 431 260 431 215 C 431 168 420 125 398 86 C 372 39 337 16 294 16 C 252 16 219 36 194 75 C 185 88 181 101 181 114 L 181 333 C 210 384 251 409 304 409 Z ",[80,3484],{"id":3485,"className":3486,"d":3487},"gl3Hi1VM",[84],"M 364 378 C 364 417 327 442 288 442 C 237 442 198 412 172 351 L 172 442 L 28 431 L 28 393 C 63 393 84 390 92 384 C 100 378 104 365 104 342 L 104 79 C 104 60 101 48 94 44 C 87 40 65 38 28 38 L 28 0 L 143 3 C 185 4 228 3 271 0 L 271 38 L 247 38 C 214 38 193 41 186 46 C 179 51 176 63 176 81 L 176 232 C 176 275 184 314 199 347 C 219 390 248 412 287 413 C 277 403 272 391 272 377 C 272 346 287 331 318 331 C 345 331 364 352 364 378 Z ",[80,3489],{"id":1026,"className":3490,"d":1028},[84],[80,3492],{"id":1031,"className":3493,"d":1033},[84],[80,3495],{"id":3496,"className":3497,"d":3498},"gl3Hi1Vo",[84],"M 113 -205 C 168 -205 212 -167 243 -90 L 416 332 C 433 372 463 392 508 392 L 508 431 C 475 428 449 427 430 428 L 341 431 L 341 393 C 372 392 387 379 387 354 C 387 348 385 341 381 332 L 286 98 L 180 354 C 177 362 175 368 174 371 C 174 385 193 392 230 392 L 230 431 L 118 428 C 94 427 61 428 19 431 L 19 392 C 72 392 85 394 99 360 L 246 1 L 222 -55 C 195 -119 176 -175 113 -175 C 99 -175 87 -171 76 -164 C 99 -159 110 -144 110 -120 C 110 -90 95 -75 64 -75 C 34 -75 19 -90 19 -121 C 19 -170 64 -205 113 -205 Z ",[80,3500],{"id":3501,"className":3502,"d":3503},"gl3Hi1Vs",[84],"M 57 0 L 384 0 L 401 191 L 369 191 C 364 129 354 87 336 65 C 318 43 283 32 230 32 L 122 32 L 389 395 C 396 404 399 410 399 414 C 399 425 389 431 370 431 L 53 431 L 41 265 L 74 265 C 77 320 87 356 103 374 C 119 392 152 402 202 402 L 306 402 L 38 38 C 31 29 28 23 28 18 C 28 1 35 0 57 0 Z ",[80,3505],{"id":2050,"className":3506,"d":2052},[84],[75,3508],{"className":3509},[89],[70,3511],{"type":72},[93,3513,3518],{"className":3514,"transform":97,"dataTid":3515,"dataPageWidth":3516,"dataPageHeight":3517},[96],"pTg80iIXiPgYghiBWFJFpk8","207","42",[93,3519,3522],{"dataTid":3520,"className":3521},"gTg80iIXiPgYghiBWFJFpk8",[105],[93,3523,3525],{"transform":108,"dataTid":3524},"pVj9d0CykknstrBp5Tmiwrc",[93,3526,3529,3711,3867,4018],{"dataTid":3527,"className":3528},"gVj9d0CykknstrBp5Tmiwrc",[105],[93,3530,3532],{"transform":108,"dataTid":3531},"ppiF1Q7rtFr4yl0WbdlMDSg",[93,3533,3536,3569,3584,3599,3614,3649,3664,3706],{"dataTid":3534,"className":3535},"gpiF1Q7rtFr4yl0WbdlMDSg",[105],[93,3537,3539],{"transform":348,"dataTid":3538},"plRshvYvPUf4e2eJqLJ32JM",[93,3540,3543,3546,3550,3553,3557,3560],{"dataTid":3541,"className":3542,"fill":130,"transform":131},"glRshvYvPUf4e2eJqLJ32JM",[129],[133,3544],{"x":135,"y":135,"href":3545},"#gl3Hi1UM",[133,3547],{"x":3548,"y":135,"href":3549},"556","#gl3Hi1Vo",[133,3551],{"x":3552,"y":135,"href":1186},"1084",[133,3554],{"x":3555,"y":135,"href":3556},"1473","#gl3Hi1UY",[133,3558],{"x":3559,"y":135,"href":1174},"1917",[93,3561,3562],{"transform":139},[141,3563,3565],{"x":135,"y":143,"width":3564,"height":145},"144.44",[147,3566,3568],{"className":3567,"style":151},[150],"bytes",[93,3570,3572],{"transform":3571,"dataTid":384},"translate(28.477,7.634)",[93,3573,3575,3577],{"dataTid":387,"className":3574,"fill":130,"transform":131},[129],[133,3576],{"x":135,"y":135,"href":391},[93,3578,3579],{"transform":139},[141,3580,3581],{"x":135,"y":143,"width":396,"height":145},[147,3582,400],{"className":3583,"style":151},[150],[93,3585,3587],{"transform":3586,"dataTid":2122},"translate(40.090,7.634)",[93,3588,3590,3592],{"dataTid":2125,"className":3589,"fill":130,"transform":131},[129],[133,3591],{"x":135,"y":135,"href":2129},[93,3593,3594],{"transform":139},[141,3595,3596],{"x":135,"y":143,"width":2096,"height":145},[147,3597,2137],{"className":3598,"style":151},[150],[93,3600,3602],{"transform":3601,"dataTid":2160},"translate(48.035,7.634)",[93,3603,3605,3607],{"dataTid":2163,"className":3604,"fill":130,"transform":131},[129],[133,3606],{"x":135,"y":135,"href":2167},[93,3608,3609],{"transform":139},[141,3610,3611],{"x":135,"y":143,"width":396,"height":145},[147,3612,2175],{"className":3613,"style":151},[150],[93,3615,3618],{"transform":3616,"dataTid":3617},"translate(59.037,7.634)","pkffiPQsBaXA\u002FKUyL+aY8WI",[93,3619,3622,3625,3628,3630,3633,3637,3640],{"dataTid":3620,"className":3621,"fill":130,"transform":131},"gkffiPQsBaXA\u002FKUyL+aY8WI",[129],[133,3623],{"x":135,"y":135,"href":3624},"#gl3Hi1U0",[133,3626],{"x":3627,"y":135,"href":1194},"278",[133,3629],{"x":2970,"y":135,"href":3549},[133,3631],{"x":3632,"y":135,"href":3556},"1306",[133,3634],{"x":3635,"y":135,"href":3636},"1750","#gl3Hi1VM",[133,3638],{"x":3639,"y":135,"href":1174},"2142",[93,3641,3642],{"transform":139},[141,3643,3645],{"x":135,"y":143,"width":3644,"height":145},"158.50",[147,3646,3648],{"className":3647,"style":151},[150],"layers",[93,3650,3652],{"transform":3651,"dataTid":2160},"translate(89.377,7.634)",[93,3653,3655,3657],{"dataTid":2163,"className":3654,"fill":130,"transform":131},[129],[133,3656],{"x":135,"y":135,"href":2167},[93,3658,3659],{"transform":139},[141,3660,3661],{"x":135,"y":143,"width":396,"height":145},[147,3662,2175],{"className":3663,"style":151},[150],[93,3665,3668],{"transform":3666,"dataTid":3667},"translate(100.380,7.634)","pUZZlbu8o3tUKe2PwSv\u002FBVU",[93,3669,3672,3674,3676,3680,3684,3687,3690,3694,3697],{"dataTid":3670,"className":3671,"fill":130,"transform":131},"gUZZlbu8o3tUKe2PwSv\u002FBVU",[129],[133,3673],{"x":135,"y":135,"href":2967},[133,3675],{"x":2970,"y":135,"href":2971},[133,3677],{"x":3678,"y":135,"href":3679},"1528","#gl3Hi1QE",[133,3681],{"x":3682,"y":135,"href":3683},"1860","#gl3Hi1Uk",[133,3685],{"x":3686,"y":135,"href":3556},"2416",[133,3688],{"x":3689,"y":135,"href":1194},"2860",[133,3691],{"x":3692,"y":135,"href":3693},"3360","#gl3Hi1UU",[133,3695],{"x":3696,"y":135,"href":1174},"3916",[93,3698,3699],{"transform":139},[141,3700,3702],{"x":135,"y":143,"width":3701,"height":145},"269.38",[147,3703,3705],{"className":3704,"style":151},[150],"KV heads",[93,3707,3708],{"transform":108,"dataTid":155},[93,3709],{"dataTid":158,"className":3710,"dataHint":18},[160],[93,3712,3715],{"transform":3713,"dataTid":3714},"translate(0.000,17.039)","pUxv+OGZTOfMCKA3JKwerV4",[93,3716,3719,3733,3792,3807,3862],{"dataTid":3717,"className":3718},"gUxv+OGZTOfMCKA3JKwerV4",[105],[93,3720,3721],{"transform":3586,"dataTid":2160},[93,3722,3724,3726],{"dataTid":2163,"className":3723,"fill":130,"transform":131},[129],[133,3725],{"x":135,"y":135,"href":2167},[93,3727,3728],{"transform":139},[141,3729,3730],{"x":135,"y":143,"width":396,"height":145},[147,3731,2175],{"className":3732,"style":151},[150],[93,3734,3737],{"transform":3735,"dataTid":3736},"translate(51.093,7.634)","pzjhpPYWowxAV8+1rhNsfT8",[93,3738,3741,3743,3745,3748,3751,3754,3757,3761,3764,3767,3771,3774,3777,3780,3783],{"dataTid":3739,"className":3740,"fill":130,"transform":131},"gzjhpPYWowxAV8+1rhNsfT8",[129],[133,3742],{"x":135,"y":135,"href":3683},[133,3744],{"x":3548,"y":135,"href":3556},[133,3746],{"x":3747,"y":135,"href":1194},"1000",[133,3749],{"x":3750,"y":135,"href":3693},"1500",[133,3752],{"x":3753,"y":135,"href":3679},"2056",[133,3755],{"x":3756,"y":135,"href":3693},"2388",[133,3758],{"x":3759,"y":135,"href":3760},"2944","#gl3Hi1Uo",[133,3762],{"x":3763,"y":135,"href":1190},"3222",[133,3765],{"x":3766,"y":135,"href":3556},"4055",[133,3768],{"x":3769,"y":135,"href":3770},"4499","#gl3Hi1U8",[133,3772],{"x":3773,"y":135,"href":1174},"5055",[133,3775],{"x":3776,"y":135,"href":3760},"5449",[133,3778],{"x":3779,"y":135,"href":1178},"5727",[133,3781],{"x":3782,"y":135,"href":3770},"6227",[93,3784,3785],{"transform":139},[141,3786,3788],{"x":135,"y":143,"width":3787,"height":145},"423.94",[147,3789,3791],{"className":3790,"style":151},[150],"head dimension",[93,3793,3795],{"transform":3794,"dataTid":2160},"translate(128.150,7.634)",[93,3796,3798,3800],{"dataTid":2163,"className":3797,"fill":130,"transform":131},[129],[133,3799],{"x":135,"y":135,"href":2167},[93,3801,3802],{"transform":139},[141,3803,3804],{"x":135,"y":143,"width":396,"height":145},[147,3805,2175],{"className":3806,"style":151},[150],[93,3808,3811],{"transform":3809,"dataTid":3810},"translate(139.152,7.634)","p8KsFn8bHs7AiaPqOxGKYXs",[93,3812,3815,3818,3821,3824,3827,3830,3832,3834,3837,3840,3844,3847,3850,3853],{"dataTid":3813,"className":3814,"fill":130,"transform":131},"g8KsFn8bHs7AiaPqOxGKYXs",[129],[133,3816],{"x":135,"y":135,"href":3817},"#gl3Hi1UQ",[133,3819],{"x":3820,"y":135,"href":1194},"444",[133,3822],{"x":3823,"y":135,"href":3817},"944",[133,3825],{"x":3826,"y":135,"href":3683},"1388",[133,3828],{"x":3829,"y":135,"href":3556},"1944",[133,3831],{"x":3756,"y":135,"href":3693},[133,3833],{"x":3759,"y":135,"href":3679},[133,3835],{"x":3836,"y":135,"href":1186},"3276",[133,3838],{"x":3839,"y":135,"href":1178},"3665",[133,3841],{"x":3842,"y":135,"href":3843},"4165","#gl3Hi1Uw",[133,3845],{"x":3846,"y":135,"href":3556},"4693",[133,3848],{"x":3849,"y":135,"href":3770},"5137",[133,3851],{"x":3852,"y":135,"href":1174},"5693",[93,3854,3855],{"transform":139},[141,3856,3858],{"x":135,"y":143,"width":3857,"height":145},"380.44",[147,3859,3861],{"className":3860,"style":151},[150],"cached tokens",[93,3863,3864],{"transform":108,"dataTid":155},[93,3865],{"dataTid":158,"className":3866,"dataHint":18},[160],[93,3868,3871],{"transform":3869,"dataTid":3870},"translate(0.000,31.944)","pOdcUZ0Ai3t8mgmLmbBJMM4",[93,3872,3875,3889,3952,3967,4013],{"dataTid":3873,"className":3874},"gOdcUZ0Ai3t8mgmLmbBJMM4",[105],[93,3876,3877],{"transform":3586,"dataTid":2160},[93,3878,3880,3882],{"dataTid":2163,"className":3879,"fill":130,"transform":131},[129],[133,3881],{"x":135,"y":135,"href":2167},[93,3883,3884],{"transform":139},[141,3885,3886],{"x":135,"y":143,"width":396,"height":145},[147,3887,2175],{"className":3888,"style":151},[150],[93,3890,3892],{"transform":3735,"dataTid":3891},"pkvvBqh0psGo\u002F718pJvlHS8",[93,3893,3896,3898,3900,3902,3904,3906,3909,3913,3916,3919,3922,3925,3928,3931,3934,3937,3940,3943],{"dataTid":3894,"className":3895,"fill":130,"transform":131},"gkvvBqh0psGo\u002F718pJvlHS8",[129],[133,3897],{"x":135,"y":135,"href":3545},[133,3899],{"x":3548,"y":135,"href":3549},[133,3901],{"x":3552,"y":135,"href":1186},[133,3903],{"x":3555,"y":135,"href":3556},[133,3905],{"x":3559,"y":135,"href":1174},[133,3907],{"x":3908,"y":135,"href":3679},"2311",[133,3910],{"x":3911,"y":135,"href":3912},"2643","#gl3Hi1VE",[133,3914],{"x":3915,"y":135,"href":3556},"3199",[133,3917],{"x":3918,"y":135,"href":3636},"3643",[133,3920],{"x":3921,"y":135,"href":3679},"4035",[133,3923],{"x":3924,"y":135,"href":3556},"4367",[133,3926],{"x":3927,"y":135,"href":3624},"4811",[133,3929],{"x":3930,"y":135,"href":3556},"5089",[133,3932],{"x":3933,"y":135,"href":1190},"5533",[133,3935],{"x":3936,"y":135,"href":3556},"6366",[133,3938],{"x":3939,"y":135,"href":3770},"6810",[133,3941],{"x":3942,"y":135,"href":1186},"7366",[93,3944,3945],{"transform":139},[141,3946,3948],{"x":135,"y":143,"width":3947,"height":145},"484.69",[147,3949,3951],{"className":3950,"style":151},[150],"bytes per element",[93,3953,3955],{"transform":3954,"dataTid":2160},"translate(138.842,7.634)",[93,3956,3958,3960],{"dataTid":2163,"className":3957,"fill":130,"transform":131},[129],[133,3959],{"x":135,"y":135,"href":2167},[93,3961,3962],{"transform":139},[141,3963,3964],{"x":135,"y":143,"width":396,"height":145},[147,3965,2175],{"className":3966,"style":151},[150],[93,3968,3971],{"transform":3969,"dataTid":3970},"translate(149.844,7.634)","pydSVPlR6HocYR2of2odZsM",[93,3972,3975,3977,3979,3982,3985,3988,3991,3994,3997,4001,4004],{"dataTid":3973,"className":3974,"fill":130,"transform":131},"gydSVPlR6HocYR2of2odZsM",[129],[133,3976],{"x":135,"y":135,"href":3545},[133,3978],{"x":3548,"y":135,"href":1194},[133,3980],{"x":3981,"y":135,"href":1186},"1056",[133,3983],{"x":3984,"y":135,"href":3817},"1445",[133,3986],{"x":3987,"y":135,"href":3683},"1889",[133,3989],{"x":3990,"y":135,"href":3679},"2445",[133,3992],{"x":3993,"y":135,"href":1174},"2777",[133,3995],{"x":3996,"y":135,"href":3760},"3171",[133,3998],{"x":3999,"y":135,"href":4000},"3449","#gl3Hi1Vs",[133,4002],{"x":4003,"y":135,"href":3556},"3893",[93,4005,4006],{"transform":139},[141,4007,4009],{"x":135,"y":143,"width":4008,"height":145},"271.06",[147,4010,4012],{"className":4011,"style":151},[150],"batch size",[93,4014,4015],{"transform":108,"dataTid":155},[93,4016],{"dataTid":158,"className":4017,"dataHint":18},[160],[93,4019,4020],{"transform":108,"dataTid":155},[93,4021],{"dataTid":158,"className":4022,"dataHint":18},[160],[11,4024,4025],{},"If sequence lengths differ, replace cached tokens times batch size with the sum of the tokens kept by all sequences. These counts assume full attention in every layer, equal key and value dimensions, and no sharing of cached data between sequences or layers.",[11,4027,4028,4029,4034],{},"The head count needs care. In multi-head attention, each query head has its own key and value head. In ",[18,4030,4033],{"href":4031,"rel":4032},"https:\u002F\u002Farxiv.org\u002Fabs\u002F2305.13245",[985],"grouped-query attention",", several query heads share a key and value head. Multi-query attention uses a single key and value head for all query heads. Each query head still computes its own scores, but the shared K and V only need to be stored once.",[11,4036,4037],{},[2579,4038],{"alt":4039,"src":4040},"With 8 query heads, multi-head attention keeps 8 KV heads, grouped-query attention with 4 heads per group keeps 2, and multi-query attention keeps 1.","\u002Fassets\u002Fblog\u002Fthe-hidden-memory-cost-of-llm-inference-kv-cache-explained\u002Fkv-cache-attention-heads.svg",[11,4042,4043,4044,4047],{},"With the same layers, head dimension and precision, going from 8 KV heads to 2 cuts the cache to a quarter. A single KV head cuts it to an eighth. Read ",[1972,4045,4046],{},"num_key_value_heads"," in the model configuration rather than using the number of query heads. You cannot change a trained model's attention design just by changing a serving option.",[11,4049,4050],{},"Take a hypothetical model with 32 layers, 8 KV heads, a head dimension of 128 and a 16-bit cache. One cached token across all layers costs",[56,4052,4057,4059,4119,4122,4124],{"style":258,"className":4053,"viewBox":4054,"width":4055,"height":1892,"dataWidth":4056,"dataHeight":1893,"xmlns":66,"xmlnsXLink":67,"xmlns:h5":68},[60],"0 0 232.000 10.000","21.09090909090909em","232.000",[70,4058,73],{"type":72},[75,4060,4062,4065,4068,4073,4076,4079,4082,4087,4090,4095,4098,4101,4104,4107,4110,4113,4116],{"className":4061},[78],[80,4063],{"id":2007,"className":4064,"d":2009},[84],[80,4066],{"id":2017,"className":4067,"d":2019},[84],[80,4069],{"id":4070,"className":4071,"d":4072},"gl3Hi1RI",[84],"M 269 666 C 228 624 168 603 89 603 L 89 564 C 141 564 184 572 217 588 L 217 82 C 217 64 213 52 204 47 C 195 42 170 39 130 39 L 95 39 L 95 0 C 120 2 174 3 257 3 C 340 3 394 2 419 0 L 419 39 L 384 39 C 343 39 318 42 310 47 C 302 52 297 64 297 82 L 297 636 C 297 660 295 666 269 666 Z ",[80,4074],{"id":2022,"className":4075,"d":2024},[84],[80,4077],{"id":2027,"className":4078,"d":2029},[84],[80,4080],{"id":2042,"className":4081,"d":2044},[84],[80,4083],{"id":4084,"className":4085,"d":4086},"gl3Hi1Rk",[84],"M 250 666 C 201 666 158 650 122 618 C 86 586 68 546 68 497 C 68 456 83 419 113 386 C 120 378 142 361 179 335 C 88 288 42 227 42 153 C 42 100 64 57 107 24 C 147 -7 194 -22 249 -22 C 305 -22 354 -4 395 32 C 436 68 457 115 457 170 C 457 216 441 257 408 294 C 396 307 365 330 316 361 C 393 402 431 454 431 515 C 431 606 344 666 250 666 Z M 379 515 C 379 463 348 418 286 381 L 167 459 C 136 479 120 505 120 536 C 120 596 185 633 249 633 C 318 633 379 584 379 515 Z M 250 14 C 168 14 99 73 99 153 C 99 220 136 274 210 315 L 328 240 C 376 209 400 174 400 134 C 400 62 325 14 250 14 Z ",[80,4088],{"id":272,"className":4089,"d":274},[84],[80,4091],{"id":4092,"className":4093,"d":4094},"gl3Hi1SM",[84],"M 478 358 C 548 376 624 432 624 512 C 624 565 599 607 549 640 C 506 669 456 683 401 683 L 36 683 L 36 644 L 64 644 C 99 644 120 641 127 636 C 134 631 138 620 138 602 L 138 81 C 138 63 134 51 127 46 C 120 41 99 39 64 39 L 36 39 L 36 0 L 427 0 C 484 0 535 16 579 48 C 627 83 651 129 651 185 C 651 278 563 342 478 358 Z M 521 512 C 521 428 451 370 367 370 L 228 370 L 228 609 C 228 645 233 644 272 644 L 394 644 C 468 644 521 586 521 512 Z M 544 186 L 544 160 C 542 137 529 111 504 84 C 478 54 442 39 395 39 L 272 39 C 233 39 228 38 228 74 L 228 340 L 409 340 C 490 340 544 270 544 186 Z ",[80,4096],{"id":2897,"className":4097,"d":2899},[84],[80,4099],{"id":3429,"className":4100,"d":3431},[84],[80,4102],{"id":3444,"className":4103,"d":3446},[84],[80,4105],{"id":3454,"className":4106,"d":3456},[84],[80,4108],{"id":1026,"className":4109,"d":1028},[84],[80,4111],{"id":1031,"className":4112,"d":1033},[84],[80,4114],{"id":3496,"className":4115,"d":3498},[84],[80,4117],{"id":2050,"className":4118,"d":2052},[84],[75,4120],{"className":4121},[89],[70,4123],{"type":72},[93,4125,4129],{"className":4126,"transform":97,"dataTid":4127,"dataPageWidth":4128,"dataPageHeight":1914},[96],"pUppLcE4ykIgDReEuxL1Vtk","232",[93,4130,4133],{"dataTid":4131,"className":4132},"gUppLcE4ykIgDReEuxL1Vtk",[105],[93,4134,4136],{"transform":108,"dataTid":4135},"pJfLoMUIbTyUzof\u002FyqWpiFw",[93,4137,4140,4456],{"dataTid":4138,"className":4139},"gJfLoMUIbTyUzof\u002FyqWpiFw",[105],[93,4141,4143],{"transform":108,"dataTid":4142},"pw6rDEiYp6ykw2xQlB5Hguk",[93,4144,4147,4161,4176,4191,4206,4221,4239,4254,4273,4288,4303,4318,4333,4348,4401,4416,4451],{"dataTid":4145,"className":4146},"gw6rDEiYp6ykw2xQlB5Hguk",[105],[93,4148,4149],{"transform":348,"dataTid":2122},[93,4150,4152,4154],{"dataTid":2125,"className":4151,"fill":130,"transform":131},[129],[133,4153],{"x":135,"y":135,"href":2129},[93,4155,4156],{"transform":139},[141,4157,4158],{"x":135,"y":143,"width":2096,"height":145},[147,4159,2137],{"className":4160,"style":151},[150],[93,4162,4164],{"transform":4163,"dataTid":2160},"translate(7.944,7.634)",[93,4165,4167,4169],{"dataTid":2163,"className":4166,"fill":130,"transform":131},[129],[133,4168],{"x":135,"y":135,"href":2167},[93,4170,4171],{"transform":139},[141,4172,4173],{"x":135,"y":143,"width":396,"height":145},[147,4174,2175],{"className":4175,"style":151},[150],[93,4177,4179],{"transform":4178,"dataTid":2532},"translate(18.947,7.634)",[93,4180,4182,4184],{"dataTid":2535,"className":4181,"fill":130,"transform":131},[129],[133,4183],{"x":135,"y":135,"href":2539},[93,4185,4186],{"transform":139},[141,4187,4188],{"x":135,"y":143,"width":2096,"height":145},[147,4189,2547],{"className":4190,"style":151},[150],[93,4192,4194],{"transform":4193,"dataTid":2122},"translate(24.447,7.634)",[93,4195,4197,4199],{"dataTid":2125,"className":4196,"fill":130,"transform":131},[129],[133,4198],{"x":135,"y":135,"href":2129},[93,4200,4201],{"transform":139},[141,4202,4203],{"x":135,"y":143,"width":2096,"height":145},[147,4204,2137],{"className":4205,"style":151},[150],[93,4207,4209],{"transform":4208,"dataTid":2160},"translate(32.391,7.634)",[93,4210,4212,4214],{"dataTid":2163,"className":4211,"fill":130,"transform":131},[129],[133,4213],{"x":135,"y":135,"href":2167},[93,4215,4216],{"transform":139},[141,4217,4218],{"x":135,"y":143,"width":396,"height":145},[147,4219,2175],{"className":4220,"style":151},[150],[93,4222,4225],{"transform":4223,"dataTid":4224},"translate(43.394,7.634)","pMyyXp9wo4lgazCtD0QCWhc",[93,4226,4229,4232],{"dataTid":4227,"className":4228,"fill":130,"transform":131},"gMyyXp9wo4lgazCtD0QCWhc",[129],[133,4230],{"x":135,"y":135,"href":4231},"#gl3Hi1Rk",[93,4233,4234],{"transform":139},[141,4235,4236],{"x":135,"y":143,"width":2096,"height":145},[147,4237,3164],{"className":4238,"style":151},[150],[93,4240,4242],{"transform":4241,"dataTid":2160},"translate(51.338,7.634)",[93,4243,4245,4247],{"dataTid":2163,"className":4244,"fill":130,"transform":131},[129],[133,4246],{"x":135,"y":135,"href":2167},[93,4248,4249],{"transform":139},[141,4250,4251],{"x":135,"y":143,"width":396,"height":145},[147,4252,2175],{"className":4253,"style":151},[150],[93,4255,4258],{"transform":4256,"dataTid":4257},"translate(62.341,7.634)","pJbrcoPGm0U8c7JjAI0dsdM",[93,4259,4262,4265],{"dataTid":4260,"className":4261,"fill":130,"transform":131},"gJbrcoPGm0U8c7JjAI0dsdM",[129],[133,4263],{"x":135,"y":135,"href":4264},"#gl3Hi1RI",[93,4266,4267],{"transform":139},[141,4268,4269],{"x":135,"y":143,"width":2096,"height":145},[147,4270,4272],{"className":4271,"style":151},[150],"1",[93,4274,4276],{"transform":4275,"dataTid":2122},"translate(67.841,7.634)",[93,4277,4279,4281],{"dataTid":2125,"className":4278,"fill":130,"transform":131},[129],[133,4280],{"x":135,"y":135,"href":2129},[93,4282,4283],{"transform":139},[141,4284,4285],{"x":135,"y":143,"width":2096,"height":145},[147,4286,2137],{"className":4287,"style":151},[150],[93,4289,4291],{"transform":4290,"dataTid":4224},"translate(73.341,7.634)",[93,4292,4294,4296],{"dataTid":4227,"className":4293,"fill":130,"transform":131},[129],[133,4295],{"x":135,"y":135,"href":4231},[93,4297,4298],{"transform":139},[141,4299,4300],{"x":135,"y":143,"width":2096,"height":145},[147,4301,3164],{"className":4302,"style":151},[150],[93,4304,4306],{"transform":4305,"dataTid":2160},"translate(81.285,7.634)",[93,4307,4309,4311],{"dataTid":2163,"className":4308,"fill":130,"transform":131},[129],[133,4310],{"x":135,"y":135,"href":2167},[93,4312,4313],{"transform":139},[141,4314,4315],{"x":135,"y":143,"width":396,"height":145},[147,4316,2175],{"className":4317,"style":151},[150],[93,4319,4321],{"transform":4320,"dataTid":2122},"translate(92.288,7.634)",[93,4322,4324,4326],{"dataTid":2125,"className":4323,"fill":130,"transform":131},[129],[133,4325],{"x":135,"y":135,"href":2129},[93,4327,4328],{"transform":139},[141,4329,4330],{"x":135,"y":143,"width":2096,"height":145},[147,4331,2137],{"className":4332,"style":151},[150],[93,4334,4336],{"transform":4335,"dataTid":384},"translate(100.843,7.634)",[93,4337,4339,4341],{"dataTid":387,"className":4338,"fill":130,"transform":131},[129],[133,4340],{"x":135,"y":135,"href":391},[93,4342,4343],{"transform":139},[141,4344,4345],{"x":135,"y":143,"width":396,"height":145},[147,4346,400],{"className":4347,"style":151},[150],[93,4349,4352],{"transform":4350,"dataTid":4351},"translate(112.457,7.634)","povsfKb9Na4snf6scKTzWkk",[93,4353,4356,4358,4361,4363,4365,4368,4371,4374,4377,4380,4383,4386,4389,4392],{"dataTid":4354,"className":4355,"fill":130,"transform":131},"govsfKb9Na4snf6scKTzWkk",[129],[133,4357],{"x":135,"y":135,"href":4264},[133,4359],{"x":4360,"y":135,"href":2539},"500",[133,4362],{"x":3747,"y":135,"href":4264},[133,4364],{"x":3750,"y":135,"href":2220},[133,4366],{"x":4367,"y":135,"href":2091},"1778",[133,4369],{"x":4370,"y":135,"href":2321},"2278",[133,4372],{"x":4373,"y":135,"href":2129},"2778",[133,4375],{"x":4376,"y":135,"href":3679},"3278",[133,4378],{"x":4379,"y":135,"href":3545},"3610",[133,4381],{"x":4382,"y":135,"href":3549},"4166",[133,4384],{"x":4385,"y":135,"href":1186},"4694",[133,4387],{"x":4388,"y":135,"href":3556},"5083",[133,4390],{"x":4391,"y":135,"href":1174},"5527",[93,4393,4394],{"transform":139},[141,4395,4397],{"x":135,"y":143,"width":4396,"height":145},"370.06",[147,4398,4400],{"className":4399,"style":151},[150],"131,072 bytes",[93,4402,4404],{"transform":4403,"dataTid":384},"translate(180.643,7.634)",[93,4405,4407,4409],{"dataTid":387,"className":4406,"fill":130,"transform":131},[129],[133,4408],{"x":135,"y":135,"href":391},[93,4410,4411],{"transform":139},[141,4412,4413],{"x":135,"y":143,"width":396,"height":145},[147,4414,400],{"className":4415,"style":151},[150],[93,4417,4420],{"transform":4418,"dataTid":4419},"translate(192.257,7.634)","pQlkucuYWDn0C7fagxAkASY",[93,4421,4424,4426,4428,4430,4432,4435,4438,4442],{"dataTid":4422,"className":4423,"fill":130,"transform":131},"gQlkucuYWDn0C7fagxAkASY",[129],[133,4425],{"x":135,"y":135,"href":4264},[133,4427],{"x":4360,"y":135,"href":2129},[133,4429],{"x":3747,"y":135,"href":4231},[133,4431],{"x":3750,"y":135,"href":3679},[133,4433],{"x":4434,"y":135,"href":2967},"1832",[133,4436],{"x":4437,"y":135,"href":3760},"2610",[133,4439],{"x":4440,"y":135,"href":4441},"2888","#gl3Hi1SM",[93,4443,4444],{"transform":139},[141,4445,4447],{"x":135,"y":143,"width":4446,"height":145},"224.75",[147,4448,4450],{"className":4449,"style":151},[150],"128 KiB",[93,4452,4453],{"transform":108,"dataTid":155},[93,4454],{"dataTid":158,"className":4455,"dataHint":18},[160],[93,4457,4458],{"transform":108,"dataTid":155},[93,4459],{"dataTid":158,"className":4460,"dataHint":18},[160],[11,4462,4463],{},"Multiplying this per-token cost by the number of cached positions gives",[4465,4466,4467,4480],"table",{},[4468,4469,4470],"thead",{},[4471,4472,4473,4477],"tr",{},[4474,4475,4476],"th",{},"Cached tokens",[4474,4478,4479],{},"Cache size for one sequence",[4481,4482,4483,4492,4500,4508],"tbody",{},[4471,4484,4485,4489],{},[4486,4487,4488],"td",{},"1,024",[4486,4490,4491],{},"128 MiB",[4471,4493,4494,4497],{},[4486,4495,4496],{},"8,192",[4486,4498,4499],{},"1 GiB",[4471,4501,4502,4505],{},[4486,4503,4504],{},"32,768",[4486,4506,4507],{},"4 GiB",[4471,4509,4510,4513],{},[4486,4511,4512],{},"131,072",[4486,4514,4515],{},"16 GiB",[11,4517,4518],{},"These are calculated sizes for one sequence under the assumptions above. If eight active sequences each keep 32,768 tokens in their own cache, the cache alone needs 32 GiB, before the weights and other allocations.",[11,4520,4521],{},[2579,4522],{"alt":4523,"src":4524},"Cache size per sequence grows linearly, from 128 MiB at 1,024 tokens to 16 GiB at 131,072 tokens.","\u002Fassets\u002Fblog\u002Fthe-hidden-memory-cost-of-llm-inference-kv-cache-explained\u002Fkv-cache-growth.svg",[11,4526,4527],{},"This is the logical size of the cached data. Padding, fixed-size blocks and space reserved for later tokens can make the actual allocation larger. If the model is split across GPUs, the formula counts the cache for the whole model, not its allocation on each device.",[11,4529,4530,4531,4536],{},"For a real example, take Qwen2.5-7B-Instruct at ",[18,4532,4535],{"href":4533,"rel":4534},"https:\u002F\u002Fhuggingface.co\u002FQwen\u002FQwen2.5-7B-Instruct\u002Fblob\u002Fa09a35458c702b33eeacc393d103063234e8bc28\u002Fconfig.json",[985],"revision a09a354",". Its configuration gives 28 layers, 4 KV heads, a hidden size of 3,584 and 28 query heads. Dividing 3,584 by 28 gives a head dimension of 128. Sliding-window attention is disabled in this configuration.",[11,4538,4539],{},"At 2 bytes per cache element, that works out to 56 KiB per token. One sequence needs 56 MiB at 1,024 tokens, 448 MiB at 8,192 tokens and 1.75 GiB at 32,768 tokens. These are calculated cache sizes, not GPU measurements. They differ from the earlier table because this model has fewer layers and KV heads.",[11,4541,4542],{},"The calculation fits in a short Rust program. It uses the configuration values above and needs no extra crates, model weights or GPU.",[4544,4545,4550],"pre",{"className":4546,"code":4547,"language":4548,"meta":4549,"style":4549},"language-rust shiki shiki-themes github-dark","fn main() {\n    let layers = 28_u64;\n    let kv_heads = 4;\n    let head_dimension = 3_584 \u002F 28;\n    let bytes_per_element = 2;\n    let bytes_per_token = 2 * layers * kv_heads * head_dimension * bytes_per_element;\n    let lengths = [1_024, 8_192, 32_768];\n    let mut total_bytes = 0;\n\n    for tokens in lengths {\n        let bytes = bytes_per_token * tokens;\n        total_bytes += bytes;\n        println!(\"{tokens} tokens -> {} MiB\", bytes \u002F (1024 * 1024));\n    }\n\n    println!(\"Total -> {} MiB\", total_bytes \u002F (1024 * 1024));\n}\n","rust","",[1972,4551,4552,4569,4590,4605,4625,4640,4671,4700,4718,4725,4740,4758,4770,4802,4808,4813,4839],{"__ignoreMap":4549},[4553,4554,4557,4561,4565],"span",{"class":4555,"line":4556},"line",1,[4553,4558,4560],{"class":4559},"snl16","fn",[4553,4562,4564],{"class":4563},"svObZ"," main",[4553,4566,4568],{"class":4567},"s95oV","() {\n",[4553,4570,4572,4575,4578,4580,4584,4587],{"class":4555,"line":4571},2,[4553,4573,4574],{"class":4559},"    let",[4553,4576,4577],{"class":4567}," layers ",[4553,4579,400],{"class":4559},[4553,4581,4583],{"class":4582},"sDLfK"," 28_",[4553,4585,4586],{"class":4563},"u64",[4553,4588,4589],{"class":4567},";\n",[4553,4591,4593,4595,4598,4600,4603],{"class":4555,"line":4592},3,[4553,4594,4574],{"class":4559},[4553,4596,4597],{"class":4567}," kv_heads ",[4553,4599,400],{"class":4559},[4553,4601,4602],{"class":4582}," 4",[4553,4604,4589],{"class":4567},[4553,4606,4607,4609,4612,4614,4617,4620,4623],{"class":4555,"line":1338},[4553,4608,4574],{"class":4559},[4553,4610,4611],{"class":4567}," head_dimension ",[4553,4613,400],{"class":4559},[4553,4615,4616],{"class":4582}," 3_584",[4553,4618,4619],{"class":4559}," \u002F",[4553,4621,4622],{"class":4582}," 28",[4553,4624,4589],{"class":4567},[4553,4626,4628,4630,4633,4635,4638],{"class":4555,"line":4627},5,[4553,4629,4574],{"class":4559},[4553,4631,4632],{"class":4567}," bytes_per_element ",[4553,4634,400],{"class":4559},[4553,4636,4637],{"class":4582}," 2",[4553,4639,4589],{"class":4567},[4553,4641,4643,4645,4648,4650,4652,4655,4657,4660,4662,4664,4666,4668],{"class":4555,"line":4642},6,[4553,4644,4574],{"class":4559},[4553,4646,4647],{"class":4567}," bytes_per_token ",[4553,4649,400],{"class":4559},[4553,4651,4637],{"class":4582},[4553,4653,4654],{"class":4559}," *",[4553,4656,4577],{"class":4567},[4553,4658,4659],{"class":4559},"*",[4553,4661,4597],{"class":4567},[4553,4663,4659],{"class":4559},[4553,4665,4611],{"class":4567},[4553,4667,4659],{"class":4559},[4553,4669,4670],{"class":4567}," bytes_per_element;\n",[4553,4672,4674,4676,4679,4681,4684,4687,4689,4692,4694,4697],{"class":4555,"line":4673},7,[4553,4675,4574],{"class":4559},[4553,4677,4678],{"class":4567}," lengths ",[4553,4680,400],{"class":4559},[4553,4682,4683],{"class":4567}," [",[4553,4685,4686],{"class":4582},"1_024",[4553,4688,817],{"class":4567},[4553,4690,4691],{"class":4582},"8_192",[4553,4693,817],{"class":4567},[4553,4695,4696],{"class":4582},"32_768",[4553,4698,4699],{"class":4567},"];\n",[4553,4701,4703,4705,4708,4711,4713,4716],{"class":4555,"line":4702},8,[4553,4704,4574],{"class":4559},[4553,4706,4707],{"class":4559}," mut",[4553,4709,4710],{"class":4567}," total_bytes ",[4553,4712,400],{"class":4559},[4553,4714,4715],{"class":4582}," 0",[4553,4717,4589],{"class":4567},[4553,4719,4721],{"class":4555,"line":4720},9,[4553,4722,4724],{"emptyLinePlaceholder":4723},true,"\n",[4553,4726,4728,4731,4734,4737],{"class":4555,"line":4727},10,[4553,4729,4730],{"class":4559},"    for",[4553,4732,4733],{"class":4567}," tokens ",[4553,4735,4736],{"class":4559},"in",[4553,4738,4739],{"class":4567}," lengths {\n",[4553,4741,4743,4746,4749,4751,4753,4755],{"class":4555,"line":4742},11,[4553,4744,4745],{"class":4559},"        let",[4553,4747,4748],{"class":4567}," bytes ",[4553,4750,400],{"class":4559},[4553,4752,4647],{"class":4567},[4553,4754,4659],{"class":4559},[4553,4756,4757],{"class":4567}," tokens;\n",[4553,4759,4761,4764,4767],{"class":4555,"line":4760},12,[4553,4762,4763],{"class":4567},"        total_bytes ",[4553,4765,4766],{"class":4559},"+=",[4553,4768,4769],{"class":4567}," bytes;\n",[4553,4771,4773,4776,4778,4782,4785,4788,4791,4794,4796,4799],{"class":4555,"line":4772},13,[4553,4774,4775],{"class":4563},"        println!",[4553,4777,1227],{"class":4567},[4553,4779,4781],{"class":4780},"sU2Wk","\"{tokens} tokens -> {} MiB\"",[4553,4783,4784],{"class":4567},", bytes ",[4553,4786,4787],{"class":4559},"\u002F",[4553,4789,4790],{"class":4567}," (",[4553,4792,4793],{"class":4582},"1024",[4553,4795,4654],{"class":4559},[4553,4797,4798],{"class":4582}," 1024",[4553,4800,4801],{"class":4567},"));\n",[4553,4803,4805],{"class":4555,"line":4804},14,[4553,4806,4807],{"class":4567},"    }\n",[4553,4809,4811],{"class":4555,"line":4810},15,[4553,4812,4724],{"emptyLinePlaceholder":4723},[4553,4814,4816,4819,4821,4824,4827,4829,4831,4833,4835,4837],{"class":4555,"line":4815},16,[4553,4817,4818],{"class":4563},"    println!",[4553,4820,1227],{"class":4567},[4553,4822,4823],{"class":4780},"\"Total -> {} MiB\"",[4553,4825,4826],{"class":4567},", total_bytes ",[4553,4828,4787],{"class":4559},[4553,4830,4790],{"class":4567},[4553,4832,4793],{"class":4582},[4553,4834,4654],{"class":4559},[4553,4836,4798],{"class":4582},[4553,4838,4801],{"class":4567},[4553,4840,4842],{"class":4555,"line":4841},17,[4553,4843,4844],{"class":4567},"}\n",[11,4846,4847,4848,4851],{},"Save the code as ",[1972,4849,4850],{},"estimate.rs",". With Rust installed, compile and run it from the same folder.",[4544,4853,4857],{"className":4854,"code":4855,"language":4856,"meta":4549,"style":4549},"language-bash shiki shiki-themes github-dark","rustc estimate.rs -o estimate && .\u002Festimate\n","bash",[1972,4858,4859],{"__ignoreMap":4549},[4553,4860,4861,4864,4867,4870,4873,4876],{"class":4555,"line":4556},[4553,4862,4863],{"class":4563},"rustc",[4553,4865,4866],{"class":4780}," estimate.rs",[4553,4868,4869],{"class":4582}," -o",[4553,4871,4872],{"class":4780}," estimate",[4553,4874,4875],{"class":4567}," && ",[4553,4877,4878],{"class":4563},".\u002Festimate\n",[11,4880,4881],{},"Each length represents one sequence running at the same time. The program prints 56, 448 and 1,792 MiB, then their combined total of 2,296 MiB. All three lengths fit within the 32,768 positions in the linked configuration. Longer contexts require a separate setup and are outside this example.",[24,4883,4885],{"id":4884},"ways-to-shrink-or-share-the-cache","Ways to shrink or share the cache",[11,4887,4888],{},"Cache quantization stores keys and values with fewer bits, for example 8 instead of 16, which halves the memory per element. It can cost some output quality, and some kernels need extra work to read the compressed format. Test both accuracy and speed on your own workload before turning it on.",[11,4890,4891,4892,4897],{},"Paging, as in vLLM's ",[18,4893,4896],{"href":4894,"rel":4895},"https:\u002F\u002Farxiv.org\u002Fabs\u002F2309.06180",[985],"PagedAttention",", splits the cache into fixed-size blocks instead of one long reserved region per sequence. It reduces unused gaps and space set aside for tokens that have not arrived. It changes how memory is managed. The vectors needed by full attention still have to be stored.",[11,4899,4900],{},"Sliding-window attention limits how far back certain layers look. For those layers, the engine can discard positions outside the window. The number of retained tokens stops growing once the window is full. A model with both sliding and full-attention layers needs a separate count for each type.",[11,4902,4903,4904,4909],{},"Prefix caching lets requests reuse cached data for an identical beginning, such as a long system prompt, under the same model settings. An engine that shares those cache blocks can avoid storing a separate copy for every request. Private request data must still be kept isolated. The ",[18,4905,4908],{"href":4906,"rel":4907},"https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftransformers\u002Fkv_cache",[985],"Transformers cache documentation"," also covers fixed-size, growing and quantized caches.",[24,4911,4913],{"id":4912},"what-to-measure-under-load","What to measure under load",[11,4915,4916],{},"The cache shares memory with the model weights, temporary activations and working space used by GPU kernels. Activations are the intermediate results produced during a forward pass. The allocator may also hold unused memory for later work. A calculation based only on the model file leaves out these allocations as well as the cache.",[11,4918,4919],{},"Continuous batching adds new requests to a running batch as others finish. Count the token state held by all active sequences, including requests waiting between steps. The number of tokens processed in one step and the total number retained in caches are different quantities.",[11,4921,4922],{},"An open chat does not always keep a cache on the GPU. After a request ends, the server may release its cache. A later request may need to process the conversation again, unless the server can reuse a saved prefix. Count the state the engine actually keeps, rather than every conversation stored by the application.",[11,4924,4925],{},"A server sized around the average conversation can run out of cache space when a few long requests overlap. Record memory after the model loads, after a long prefill and during generation. Then repeat with several requests running together, using the range of lengths your application handles. Leave some memory free for other work.",[11,4927,4928,4929,817,4934,4937,4938,4941,4942,4945],{},"Compare what the framework reports with what the process and the device report. In ",[18,4930,4933],{"href":4931,"rel":4932},"https:\u002F\u002Fdocs.pytorch.org\u002Fdocs\u002Fstable\u002Fnotes\u002Fcuda.html#memory-management",[985],"PyTorch",[1972,4935,4936],{},"torch.cuda.memory_allocated()"," counts memory used by live tensors. ",[1972,4939,4940],{},"torch.cuda.memory_reserved()"," also includes memory the allocator keeps for later. ",[1972,4943,4944],{},"nvidia-smi"," shows GPU memory used by the process, which can include allocations outside PyTorch's allocator. These views count different things, so none is a direct measure of the cache alone.",[11,4947,4948,4949,4954],{},"Some engines reserve their GPU cache pool at startup. The reported memory can then stay almost flat as requests fill that pool. Watch the share of cache blocks in use as well. When space runs out, ",[18,4950,4953],{"href":4951,"rel":4952},"https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Fconfiguration\u002Foptimization\u002F#preemption",[985],"vLLM can pause requests and later recompute their state",". That can increase response time without a new GPU allocation or an out-of-memory error.",[11,4956,4957],{},"Run a test with several long requests at once. Record cache use, any paused or recomputed requests, and response times alongside GPU memory. Use that result to set your serving limits, then repeat the test when the model, cache precision or expected request lengths change.",[70,4959,4960],{},"html pre.shiki code .snl16, html code.shiki .snl16{--shiki-default:#F97583}html pre.shiki code .svObZ, html code.shiki .svObZ{--shiki-default:#B392F0}html pre.shiki code .s95oV, html code.shiki .s95oV{--shiki-default:#E1E4E8}html pre.shiki code .sDLfK, html code.shiki .sDLfK{--shiki-default:#79B8FF}html pre.shiki code .sU2Wk, html code.shiki .sU2Wk{--shiki-default:#9ECBFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":4549,"searchDepth":4571,"depth":4571,"links":4962},[4963,4964,4965,4966,4967,4968],{"id":26,"depth":4571,"text":27},{"id":47,"depth":4571,"text":48},{"id":2588,"depth":4571,"text":2589},{"id":2622,"depth":4571,"text":2623},{"id":4884,"depth":4571,"text":4885},{"id":4912,"depth":4571,"text":4913},"\u002Fassets\u002Fblog\u002Fthe-hidden-memory-cost-of-llm-inference-kv-cache-explained\u002Fcover.png","2026-10-10","Calculate KV cache memory and understand how context length, architecture and concurrency affect serving capacity.","md",false,"concept","en",{},null,"\u002Fblog\u002Fthe-hidden-memory-cost-of-llm-inference-kv-cache-explained","13 min read",{"title":5,"description":4971},{"loc":4978,"images":4982},[4983,4984,4985,4986],{"loc":2582},{"loc":2616},{"loc":4040},{"loc":4524},"blog\u002Fthe-hidden-memory-cost-of-llm-inference-kv-cache-explained",[4989,4990,4991],"KV cache","LLM inference","memory","6LuG1czn8n0loPi8SuqJAJEhF0ZwTpu_yqRjWyfuGhM",[],{"left":4995,"top":4995,"width":4996,"height":4996,"rotate":4995,"vFlip":4973,"hFlip":4973,"body":4997},0,24,"\u003Cpath fill=\"currentColor\" d=\"M12 .297c-6.63 0-12 5.373-12 12c0 5.303 3.438 9.8 8.205 11.385c.6.113.82-.258.82-.577c0-.285-.01-1.04-.015-2.04c-3.338.724-4.042-1.61-4.042-1.61C4.422 18.07 3.633 17.7 3.633 17.7c-1.087-.744.084-.729.084-.729c1.205.084 1.838 1.236 1.838 1.236c1.07 1.835 2.809 1.305 3.495.998c.108-.776.417-1.305.76-1.605c-2.665-.3-5.466-1.332-5.466-5.93c0-1.31.465-2.38 1.235-3.22c-.135-.303-.54-1.523.105-3.176c0 0 1.005-.322 3.3 1.23c.96-.267 1.98-.399 3-.405c1.02.006 2.04.138 3 .405c2.28-1.552 3.285-1.23 3.285-1.23c.645 1.653.24 2.873.12 3.176c.765.84 1.23 1.91 1.23 3.22c0 4.61-2.805 5.625-5.475 5.92c.42.36.81 1.096.81 2.22c0 1.606-.015 2.896-.015 3.286c0 .315.21.69.825.57C20.565 22.092 24 17.592 24 12.297c0-6.627-5.373-12-12-12\"\u002F>",{"left":4995,"top":4995,"width":4996,"height":4996,"rotate":4995,"vFlip":4973,"hFlip":4973,"body":4999,"hidden":4723},"\u003Cpath fill=\"currentColor\" d=\"M20.447 20.452h-3.554v-5.569c0-1.328-.027-3.037-1.852-3.037c-1.853 0-2.136 1.445-2.136 2.939v5.667H9.351V9h3.414v1.561h.046c.477-.9 1.637-1.85 3.37-1.85c3.601 0 4.267 2.37 4.267 5.455v6.286zM5.337 7.433a2.06 2.06 0 0 1-2.063-2.065a2.064 2.064 0 1 1 2.063 2.065m1.782 13.019H3.555V9h3.564zM22.225 0H1.771C.792 0 0 .774 0 1.729v20.542C0 23.227.792 24 1.771 24h20.451C23.2 24 24 23.227 24 22.271V1.729C24 .774 23.2 0 22.222 0z\"\u002F>",{"left":4995,"top":4995,"width":4996,"height":4996,"rotate":4995,"vFlip":4973,"hFlip":4973,"body":5001},"\u003Cpath fill=\"currentColor\" d=\"M5.202 2.857C7.954 4.922 10.913 9.11 12 11.358c1.087-2.247 4.046-6.436 6.798-8.501C20.783 1.366 24 .213 24 3.883c0 .732-.42 6.156-.667 7.037c-.856 3.061-3.978 3.842-6.755 3.37c4.854.826 6.089 3.562 3.422 6.299c-5.065 5.196-7.28-1.304-7.847-2.97c-.104-.305-.152-.448-.153-.327c0-.121-.05.022-.153.327c-.568 1.666-2.782 8.166-7.847 2.97c-2.667-2.737-1.432-5.473 3.422-6.3c-2.777.473-5.899-.308-6.755-3.369C.42 10.04 0 4.615 0 3.883c0-3.67 3.217-2.517 5.202-1.026\"\u002F>",{"left":4995,"top":4995,"width":4996,"height":4996,"rotate":4995,"vFlip":4973,"hFlip":4973,"body":5003},"\u003Cpath fill=\"currentColor\" d=\"M14.234 10.162L22.977 0h-2.072l-7.591 8.824L7.251 0H.258l9.168 13.343L.258 24H2.33l8.016-9.318L16.749 24h6.993zm-2.837 3.299l-.929-1.329L3.076 1.56h3.182l5.965 8.532l.929 1.329l7.754 11.09h-3.182z\"\u002F>",1793168446774]