[{"data":1,"prerenderedAt":451},["ShallowReactive",2],{"blog-\u002Fblog\u002Fthe-hidden-cost-of-observability-cardinality-sampling-and-telemetry-at-scale\u002F":3,"blog-translation-\u002Fblog\u002Fthe-hidden-cost-of-observability-cardinality-sampling-and-telemetry-at-scale\u002F":423,"blog-series-\u002Fblog\u002Fthe-hidden-cost-of-observability-cardinality-sampling-and-telemetry-at-scale\u002F":440,"i-simple-icons:github":441,"i-simple-icons:linkedin":445,"i-simple-icons:bluesky":447,"i-simple-icons:x":449},{"id":4,"title":5,"author":6,"body":7,"cover":415,"date":416,"description":417,"extension":418,"featured":419,"kind":420,"language":421,"meta":422,"modified":423,"navigation":424,"path":425,"published":424,"readingTime":426,"seo":427,"seriesKey":423,"seriesOrder":423,"seriesTitle":423,"sitemap":428,"stem":434,"tags":435,"translationKey":423,"__hash__":439},"blog\u002Fblog\u002Fthe-hidden-cost-of-observability-cardinality-sampling-and-telemetry-at-scale.md","The hidden cost of observability: cardinality, sampling and telemetry at scale","Lucas Jahier",{"type":8,"value":9,"toc":406},"minimark",[10,14,21,26,29,32,36,39,95,98,102,105,166,169,178,184,200,203,285,288,292,295,298,301,307,316,320,323,326,342,345,348,362,365,369,372,378,381,384,388,402],[11,12,13],"p",{},"The cost of observability often comes from instrumentation choices made one span, one label, and one log line at a time. This article discusses where the volume comes from, how cardinality and sampling affect it, and strategies to reduce costs without losing crucial evidence during an incident.",[11,15,16],{},[17,18],"img",{"alt":19,"src":20},"Telemetry volume as the product of per-request data, rate, replicas, and retention","\u002Fassets\u002Fblog\u002Fthe-hidden-cost-of-observability-cardinality-sampling-and-telemetry-at-scale\u002Ftelemetry-volume.svg",[22,23,25],"h2",{"id":24},"small-choices-lead-to-infrastructure-decisions","Small choices lead to infrastructure decisions",[11,27,28],{},"A service can emit very little data per request and still produce a large storage bill. Spans, log lines, and metric points are multiplied by request rate, replica count, and retention period. Each factor looks harmless on its own their product is what you pay for.",[11,30,31],{},"Growth is not only a matter of bytes. Each new metric dimension can multiply the number of series. Each new indexed field adds storage and query work. The question to ask of every additional record is how does it improve your ability to operate the service?",[22,33,35],{"id":34},"costs-vary-by-signal","Costs vary by signal",[11,37,38],{},"Costs follow the shape of each signal, so inventory them signal by signal.",[40,41,42,58],"table",{},[43,44,45],"thead",{},[46,47,48,52,55],"tr",{},[49,50,51],"th",{},"Signal",[49,53,54],{},"What drives cost",[49,56,57],{},"Typical blind spot",[59,60,61,73,84],"tbody",{},[46,62,63,67,70],{},[64,65,66],"td",{},"Logs",[64,68,69],{},"Ingestion, indexing, retention",[64,71,72],{},"Verbose payloads and debug lines left on",[46,74,75,78,81],{},[64,76,77],{},"Traces",[64,79,80],{},"Span count, attributes per span",[64,82,83],{},"Repeated low-value internal spans",[46,85,86,89,92],{},[64,87,88],{},"Metrics",[64,90,91],{},"Active time series, sample frequency",[64,93,94],{},"Unbounded label values",[11,96,97],{},"A billing dashboard alone cannot tell you whether growth came from more traffic, a new debug statement or an unbounded label shipped in the latest release. Build the inventory around signals and workloads, so each increase can be traced back to a change.",[22,99,101],{"id":100},"cardinality-multiplies","Cardinality multiplies",[11,103,104],{},"A metric series is identified by its full set of dimension values. When several dimensions vary independently, the number of series is the product of their distinct values. Tenant identifiers, request identifiers, and raw URLs are common sources of an explosion in the number of series.",[40,106,107,120],{},[43,108,109],{},[46,110,111,114,117],{},[49,112,113],{},"Attribute",[49,115,116],{},"Typical use",[49,118,119],{},"Main concern",[59,121,122,133,144,155],{},[46,123,124,127,130],{},[64,125,126],{},"HTTP method",[64,128,129],{},"Aggregate metric dimension",[64,131,132],{},"Usually bounded",[46,134,135,138,141],{},[64,136,137],{},"Route template",[64,139,140],{},"Endpoint latency grouping",[64,142,143],{},"Keep templates stable",[46,145,146,149,152],{},[64,147,148],{},"Request identifier",[64,150,151],{},"Trace or log lookup",[64,153,154],{},"Unbounded metric series",[46,156,157,160,163],{},[64,158,159],{},"Customer email",[64,161,162],{},"Rarely appropriate in telemetry",[64,164,165],{},"Privacy and cardinality",[11,167,168],{},"Take one HTTP latency metric with 5 methods, 40 route templates, 5 status classes, and 20 pods: 5 × 40 × 5 × 20 = 20,000 series. Add a request identifier and every request creates a new series, at 100 requests per second, that is 100 × 86,400 = 8,640,000 new series per day.",[170,171,172],"blockquote",{},[11,173,174],{},[175,176,177],"em",{},"Note: This represents distinct series created over a day, not necessarily concurrently active series. Billing depends on the backend's handling of series churn and ingestion.",[11,179,180],{},[17,181],{"alt":182,"src":183},"Cardinality multiplies, it does not add","\u002Fassets\u002Fblog\u002Fthe-hidden-cost-of-observability-cardinality-sampling-and-telemetry-at-scale\u002Fcardinality-multiplication.svg",[11,185,186,187,191,192,199],{},"Aggregate HTTP metrics on route templates such as ",[188,189,190],"code",{},"\u002Forders\u002F{id}",", never on raw paths. Identifiers needed for individual investigations belong in a signal and a backend built for them, typically traces or logs. The OpenTelemetry ",[193,194,198],"a",{"href":195,"rel":196},"https:\u002F\u002Fopentelemetry.io\u002Fdocs\u002Fspecs\u002Fotel\u002Fmetrics\u002Fdata-model\u002F",[197],"nofollow","metrics data model"," describes how metric identity and aggregation work.",[11,201,202],{},"When request identifiers are already attached to metric data points, the OpenTelemetry Collector can remove them before export:",[204,205,210],"pre",{"className":206,"code":207,"language":208,"meta":209,"style":209},"language-yaml shiki shiki-themes github-dark","processors:\n  transform\u002Fdrop_request_id:\n    error_mode: ignore\n    metric_statements:\n      - context: datapoint\n        statements:\n          - delete_key(attributes, \"request_id\")\n","yaml","",[188,211,212,225,233,246,254,268,276],{"__ignoreMap":209},[213,214,217,221],"span",{"class":215,"line":216},"line",1,[213,218,220],{"class":219},"s4JwU","processors",[213,222,224],{"class":223},"s95oV",":\n",[213,226,228,231],{"class":215,"line":227},2,[213,229,230],{"class":219},"  transform\u002Fdrop_request_id",[213,232,224],{"class":223},[213,234,236,239,242],{"class":215,"line":235},3,[213,237,238],{"class":219},"    error_mode",[213,240,241],{"class":223},": ",[213,243,245],{"class":244},"sU2Wk","ignore\n",[213,247,249,252],{"class":215,"line":248},4,[213,250,251],{"class":219},"    metric_statements",[213,253,224],{"class":223},[213,255,257,260,263,265],{"class":215,"line":256},5,[213,258,259],{"class":223},"      - ",[213,261,262],{"class":219},"context",[213,264,241],{"class":223},[213,266,267],{"class":244},"datapoint\n",[213,269,271,274],{"class":215,"line":270},6,[213,272,273],{"class":219},"        statements",[213,275,224],{"class":223},[213,277,279,282],{"class":215,"line":278},7,[213,280,281],{"class":223},"          - ",[213,283,284],{"class":244},"delete_key(attributes, \"request_id\")\n",[11,286,287],{},"This removes the attribute before backend ingestion, but it does not undo cardinality or aggregation work already performed by the SDK. Avoiding unbounded attributes during instrumentation is preferable.",[22,289,291],{"id":290},"sampling-strategies","Sampling strategies",[11,293,294],{},"Head sampling decides at the start of a request whether to collect data or not. It limits work and export volume close to the source. A random policy gives broad coverage, but it can discard rare failures that only become visible later in the request.",[11,296,297],{},"Tail sampling decides once enough of the trace has been collected to evaluate a policy. It can keep errors and slow requests, at the price of buffering, trace-aware routing, and decision latency in the collector tier. It cannot recover spans already dropped upstream, so the two stages must be designed together.",[11,299,300],{},"Tail sampling reduces the volume sent to storage, but does not necessarily reduce instrumentation or collection costs. Collectors still need enough memory and processing capacity to receive and evaluate traces before discarding them.",[11,302,303],{},[17,304],{"alt":305,"src":306},"Head sampling in the SDK, tail sampling in the collector, then the backend","\u002Fassets\u002Fblog\u002Fthe-hidden-cost-of-observability-cardinality-sampling-and-telemetry-at-scale\u002Fhead-tail-sampling.svg",[11,308,309,310,315],{},"A retained set biased toward errors is useful for debugging, but it is not a representative latency distribution. Compute rates and service-level indicators from measurements collected for that purpose, usually metrics. The OpenTelemetry ",[193,311,314],{"href":312,"rel":313},"https:\u002F\u002Fopentelemetry.io\u002Fdocs\u002Fconcepts\u002Fsampling\u002F",[197],"sampling guide"," discusses these trade-offs.",[22,317,319],{"id":318},"retention-policies-should-be-question-driven","Retention policies should be question-driven",[11,321,322],{},"Different questions need different time horizons. Recent traces support detailed incident investigation. Aggregated metrics support capacity planning over longer periods. Audit events often carry their own regulatory requirements, separate from ordinary debug logs. Define retention by purpose, then verify how the backend actually deletes data. Time-based policies often run asynchronously. An expiry setting is not a guarantee that every byte disappears the instant its timestamp crosses the threshold.",[11,324,325],{},"Filter Early, Aggregate to the Questions You Actually Ask:",[327,328,329,333,336,339],"ul",{},[330,331,332],"li",{},"Remove attributes that have no investigative value.",[330,334,335],{},"Drop predictable noise before export or indexing, where it is most expensive.",[330,337,338],{},"Aggregate measurements at the granularity your operational questions use.",[330,340,341],{},"Keep a documented path to raise detail temporarily during a focused investigation.",[11,343,344],{},"A pipeline change can silently change what a dashboard means. Review transformations alongside instrumentation changes, and keep enough provenance to know which policy produced a given dataset.",[11,346,347],{},"A cost reduction succeeds only if the remaining data still supports the investigations you need. Track the indicators that show whether it does:",[327,349,350,353,356,359],{},[330,351,352],{},"Context propagation coverage across services.",[330,354,355],{},"Export failures and dropped data in the pipeline.",[330,357,358],{},"Telemetry missing resource identity (service, version, environment).",[330,360,361],{},"Availability of relevant traces for known incidents.",[11,363,364],{},"Then test it directly. Inject a controlled failure and ask an engineer to find the affected operation, identify the dependency, and explain the error. That exercise measures usefulness far better than a drop in stored volume.",[22,366,368],{"id":367},"balance-cost-against-investigative-value","Balance cost against investigative value",[11,370,371],{},"Start with the largest contributors in your inventory. Replace an unbounded metric label, remove redundant log payloads, collapse repeated low-value spans. Measure the effect of each change on its own before adjusting sampling and retention together by changing both at once hides which one removed the evidence.",[11,373,374],{},[17,375],{"alt":376,"src":377},"Five-step optimisation loop","\u002Fassets\u002Fblog\u002Fthe-hidden-cost-of-observability-cardinality-sampling-and-telemetry-at-scale\u002Ffive-step-optimisation-loop.svg",[11,379,380],{},"Keep a short comparison period where you can. If a policy removes evidence needed for an investigation, revisit it with that concrete case, not with a general preference for more data.",[11,382,383],{},"Explicit Budgets, Explicit Investigative Goals:\nA sustainable observability system states both what it may cost and what it must be able to explain. The work is to manage the relationship between the two enough context to explain the behavior that matters, at a delivery and storage cost the platform can sustain.",[22,385,387],{"id":386},"sources","Sources",[327,389,390,396],{},[330,391,392],{},[193,393,395],{"href":195,"rel":394},[197],"OpenTelemetry, Metrics data model",[330,397,398],{},[193,399,401],{"href":312,"rel":400},[197],"OpenTelemetry, Sampling",[403,404,405],"style",{},"html pre.shiki code .s4JwU, html code.shiki .s4JwU{--shiki-default:#85E89D}html pre.shiki code .s95oV, html code.shiki .s95oV{--shiki-default:#E1E4E8}html pre.shiki code .sU2Wk, html code.shiki .sU2Wk{--shiki-default:#9ECBFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":209,"searchDepth":227,"depth":227,"links":407},[408,409,410,411,412,413,414],{"id":24,"depth":227,"text":25},{"id":34,"depth":227,"text":35},{"id":100,"depth":227,"text":101},{"id":290,"depth":227,"text":291},{"id":318,"depth":227,"text":319},{"id":367,"depth":227,"text":368},{"id":386,"depth":227,"text":387},"\u002Fassets\u002Fblog\u002Fthe-hidden-cost-of-observability-cardinality-sampling-and-telemetry-at-scale\u002Fcover.png","2025-10-16","Control telemetry costs through cardinality limits, sampling, retention and measures of investigative usefulness.","md",false,"concept","en",{},null,true,"\u002Fblog\u002Fthe-hidden-cost-of-observability-cardinality-sampling-and-telemetry-at-scale","6 min read",{"title":5,"description":417},{"loc":425,"images":429},[430,431,432,433],{"loc":20},{"loc":183},{"loc":306},{"loc":377},"blog\u002Fthe-hidden-cost-of-observability-cardinality-sampling-and-telemetry-at-scale",[436,437,438],"observability","sampling","cardinality","uY60wl0IMJ4mp8DwQorx1ax6cLFO2KP7ohSWHcf6lmY",[],{"left":442,"top":442,"width":443,"height":443,"rotate":442,"vFlip":419,"hFlip":419,"body":444},0,24,"\u003Cpath fill=\"currentColor\" d=\"M12 .297c-6.63 0-12 5.373-12 12c0 5.303 3.438 9.8 8.205 11.385c.6.113.82-.258.82-.577c0-.285-.01-1.04-.015-2.04c-3.338.724-4.042-1.61-4.042-1.61C4.422 18.07 3.633 17.7 3.633 17.7c-1.087-.744.084-.729.084-.729c1.205.084 1.838 1.236 1.838 1.236c1.07 1.835 2.809 1.305 3.495.998c.108-.776.417-1.305.76-1.605c-2.665-.3-5.466-1.332-5.466-5.93c0-1.31.465-2.38 1.235-3.22c-.135-.303-.54-1.523.105-3.176c0 0 1.005-.322 3.3 1.23c.96-.267 1.98-.399 3-.405c1.02.006 2.04.138 3 .405c2.28-1.552 3.285-1.23 3.285-1.23c.645 1.653.24 2.873.12 3.176c.765.84 1.23 1.91 1.23 3.22c0 4.61-2.805 5.625-5.475 5.92c.42.36.81 1.096.81 2.22c0 1.606-.015 2.896-.015 3.286c0 .315.21.69.825.57C20.565 22.092 24 17.592 24 12.297c0-6.627-5.373-12-12-12\"\u002F>",{"left":442,"top":442,"width":443,"height":443,"rotate":442,"vFlip":419,"hFlip":419,"body":446,"hidden":424},"\u003Cpath fill=\"currentColor\" d=\"M20.447 20.452h-3.554v-5.569c0-1.328-.027-3.037-1.852-3.037c-1.853 0-2.136 1.445-2.136 2.939v5.667H9.351V9h3.414v1.561h.046c.477-.9 1.637-1.85 3.37-1.85c3.601 0 4.267 2.37 4.267 5.455v6.286zM5.337 7.433a2.06 2.06 0 0 1-2.063-2.065a2.064 2.064 0 1 1 2.063 2.065m1.782 13.019H3.555V9h3.564zM22.225 0H1.771C.792 0 0 .774 0 1.729v20.542C0 23.227.792 24 1.771 24h20.451C23.2 24 24 23.227 24 22.271V1.729C24 .774 23.2 0 22.222 0z\"\u002F>",{"left":442,"top":442,"width":443,"height":443,"rotate":442,"vFlip":419,"hFlip":419,"body":448},"\u003Cpath fill=\"currentColor\" d=\"M5.202 2.857C7.954 4.922 10.913 9.11 12 11.358c1.087-2.247 4.046-6.436 6.798-8.501C20.783 1.366 24 .213 24 3.883c0 .732-.42 6.156-.667 7.037c-.856 3.061-3.978 3.842-6.755 3.37c4.854.826 6.089 3.562 3.422 6.299c-5.065 5.196-7.28-1.304-7.847-2.97c-.104-.305-.152-.448-.153-.327c0-.121-.05.022-.153.327c-.568 1.666-2.782 8.166-7.847 2.97c-2.667-2.737-1.432-5.473 3.422-6.3c-2.777.473-5.899-.308-6.755-3.369C.42 10.04 0 4.615 0 3.883c0-3.67 3.217-2.517 5.202-1.026\"\u002F>",{"left":442,"top":442,"width":443,"height":443,"rotate":442,"vFlip":419,"hFlip":419,"body":450},"\u003Cpath fill=\"currentColor\" d=\"M14.234 10.162L22.977 0h-2.072l-7.591 8.824L7.251 0H.258l9.168 13.343L.258 24H2.33l8.016-9.318L16.749 24h6.993zm-2.837 3.299l-.929-1.329L3.076 1.56h3.182l5.965 8.532l.929 1.329l7.754 11.09h-3.182z\"\u002F>",1793168446833]