企业官网建设流程全解析

1. 支持OpenTelemetry的前端系统和后端系统

OpenTelemetry(简称OTel)是I/T系统监控的开放标准。它的目的在于避免应用系统或者平台与特定的监控后端系统(存储监控数据，展现UI，分析，后处理(包括预警、自动恢复等))的硬绑定。这样，一旦应用系统或者平台实现了OpenTelemetry，就可以随意切换各种后端系统，或者同时使用多种后端系统。这就给用户带来了极大的方便。

所以OpenTelemetry社区虽然产出了大量的工具，都不是为了替换已有的后端系统。而各种常见的后端系统都积极地接入了OpenTelemetry，凭借其核心能力(UI、分析、后处理)而不是协议垄断来进行积极地竞争。而监控的前端系统(agent之类)则被OpenTelemetry进行大浪淘沙般地清洗，很多前端系统由于不兼容OpenTelemetry会被逐渐淘汰。目前OpenTelemetry领域最受欢迎的前端有很多，比如：

• OpenTelemetry Collector：功能强大，可以收集各种信息，进行各种转换。
• 各种编程语言的OpenTelemetry Agent：很多编程语言支持自动注入，即不需要OTLP修改代码就能收集OpenTelemetry数据(Metrics,Traces,Logs,Profiles...)
• OpenTelemetry SDK和OTLP(OpenTelemetry Protocol)协议：OpenTelemetry的编程基础，需要侵入式编程，适于大公司、I/T产品提供商和专业人士等。
• OpenTelemetry Operator：运行在Kubernetes集群上的工具，支持自动注入。用于自动支持监控Kubernetes平台和应用。
• OpenTelemetry eBPF Instrumentation (OBI)：支持零侵入式注入。OpenTelemetry社区和第三方都有很多工具。
• 第三方平台提供的OpenTelemetry支持，比如Spring 3。

而支持OpenTelemetry的后端系统，一般指的是原生接入(可以直接接入原生的OTLP协议的系统)，一些可以通过OpenTelemetry Collector的exporter完美集成的系统也勉强算OpenTelemetry的后端系统。我这篇文章讨论的是作为应用性能监控(APM)的OpenTelemetry后端系统。比如主流的商业APM都原生支持OpenTelemetry，包括几十个大品牌，这里就不赘述。支持OpenTelemetry的开源APM系统或者后端系统产品也非常多。我在这篇文章会介绍几个。

2. Prometheus, Jaeger, Elasticsearch

Prometheus,Jaeger,Elasticsearch分别是开源社区metrics(时序数据库)、traces和logs的明星产品。从OpenTelemetry诞生时就很好地支持Jaeger和Prometheus。

Prometheus过去一直是通过OpenTelemetry Collector的Prometheus exporter或者Prometheus remote write exporter来接入OpenTelemetry数据的(当然Prometheus也可以通过Prometheus receiver或者Prometheus remote write receiver把原生的OTLP数据送给OpenTelemetry Collector)。这个运行效果也非常好。而较新版本的Prometheus可以直接接受OTLP数据。在启动的命令行使用--enable-feature参数指定即可：

./prometheus --enable-feature=otlp-write-receiver,otlp-deltatocumulative --config.file="prometheus.yml"

下载Prometheus二进制的地址是：https://github.com/prometheus/prometheus/releases。找一个最新版下载即可。关于Prometheus的配置，网上有太多文档，此处就不赘述。至于UI，一般都是选大名鼎鼎的Grafana，其实设计基于Prometheus的仪表盘(Dashboard)的要点还是Prometheus的查询语言，叫做PromQL。这个需要好好学习一下，网上文档非常多。本文后面讲Grafana LGTM时还会讲一些Prometheus和Grafana的内容。

Jaeger很早就支持原生的OTLP。Jaeger不像Prometheus那样只有一个二进制文件，Jaeger有很多组件。使用Jaeger最简单的办法是运行一个Docker容器。使用如下的命令：

docker run --rm --name jaeger \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ -p 5778:5778 \ -p 9411:9411 \ jaegertracing/jaeger:2.15.1

以下是一个典型的Jaeger Trace dashboard(从查询结果点击其中一个Trace)：

这里我只展开了开始3层的调用关系。通过Trace的UI，用户可以看到父子和兄弟span的调用关系，可以从时长来分析消耗和瓶颈，从Span的标签来分析调用的来源。各种Trace后端都有类似的界面。

相对于Traces和Metrics，OpenTelemetry在较晚时间才对Logs提供全面的支持，成熟度相对较低，而OpenTelemetry对Profiles的支持还在测试版阶段。目前前端系统通过OTLP协议提供Logs和Profiles的场景还比较少。对于Elasticsearch来说，通常不是让OpenTelemetry前端通过OTLP协议直接写入 Elasticsearch，而是采用 OpenTelemetry Collector 作为中间层，使用Elasticsearch exporter来写入Elasticsearch。

下载和安装一个Elasticsearch本地运行环境的方法是用以下的命令：

curl -fsSL https://elastic.co/start-local | sh

同时会安装Elasticsearch的UI工具：Kibana。

这个Prometheus+Jaeger+Elasticsearch的解决方案，貌似精英尽出，各尽其才，其实还是有很多不足之处。第一个问题是集成度不高，各自为战。比如要用一个Dashboard展现Metrics,Traces和Logs，就不好做。第二个问题是对于一些超小的部门级应用，维护3套监控后端系统，维护的成本稍微有点高。

第三个问题是Metrics领域老大难，就是高基数(high cardinality)带来的麻烦，尤其是对于Prometheus。OpenTelemetry语义规范(Semantic Conventions)定义了不少Resource attributes是潜在高基数的，比如service.instance.id,k8s.pod.uid,container.id之类，在一个活跃的系统，如果数据保持的时间(RetentionPolicy)较长，Prometheus的内存过大，可能要爆掉。这个问题不能使用Thanos之类的系统工具来解决。

应对高基数问题的解决办法之一是使用新型的metrics store，比如目前流行的列数据库ClickHouse。这样就可以大大缓解问题，在同样的条件下，监控系统不会崩溃。但如果用户想把request.id之类的变化无穷且数量极多的东西作为metrics的label，依然是巨大的错误。这会导致存储空间快速增长，性能严重下降。所以设计上的改变才能真正解决这一问题。OpenTelemetry的解决方案是给一些metrics附加其对应的trace span的引用(trace id,span id)，称作Exemplar。把高基数的label从metrics中拿掉，放到trace，作为span的label。这个做法在很多OpenTelemetry的Agent里面已经实现。而Grafana等UI工具也能展现Exemplar。我在后面章节的Grafana LGTM里会进一步介绍。

3. Grafana LGTM

Grafana LGTM是Grafana提供的一套全面的OpenTelemetry后端系统的解决方案，可以处理OpenTelemetry Metrics，Traces，Logs，Profiles，还自带OpenTelemetry OBI工具。对于熟悉传统监控工具的工程师来说，LGTM可谓既简单又好用。

LGTM既是“Looks Good To Me”的缩写，也是其中4个组件的缩写，包括：

• Loki(L) - 日志存储
• Grafana(G) - 统一界面
• Tempo(T) - Traces的存储
• Mimir(M) - Metrics的存储，兼容Prometheus

其实，除了这4个组件，LGTM还有一些其它组件，包括：

• OpenTelemetry Collector
• Beyla - 支持 OpenTelemetry eBPF Instrumentation (OBI)
• Grafana Pyroscope - 支持 Profiling

可以在几分钟内安装好一个最小型的Grafana LGTM，可以供一个部门级的小型非关键系统使用。以下是安装命令：

cd “给LGTM创建的目录” wget https://raw.githubusercontent.com/grafana/docker-otel-lgtm/main/run-lgtm.sh

运行也非常简单，以下是命令：

cd “给LGTM创建的目录” ENABLE_OBI=true OTEL_EBPF_OPEN_PORT=8080,8000,28080 ./run-lgtm.sh

这里我用LGTM自带的OpenTelemetry OBI工具做零侵入监控。如果不需要的话，直接运行“./run-lgtm.sh”即可。

cd “给LGTM创建的目录” ./run-lgtm.sh

熟悉Prometheus和Grafana的人很多，设计相关的Dashboard的做法也很成熟。我写了一个小的Dashboard，见下图：

下面是该dashboard的源码：

{ "annotations": { "list": [ { "builtIn": 1, "datasource": { "type": "grafana", "uid": "-- Grafana --" }, "enable": true, "hide": true, "iconColor": "rgba(0, 211, 255, 1)", "name": "Annotations & Alerts", "type": "dashboard" } ] }, "editable": true, "fiscalYearStartMonth": 0, "graphTooltip": 0, "id": 4, "links": [], "panels": [ { "datasource": { "type": "prometheus", "uid": "prometheus" }, "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisBorderShow": false, "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "barWidthFactor": 0.6, "drawStyle": "line", "fillOpacity": 0, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "showValues": false, "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": 0 }, { "color": "red", "value": 80 } ] }, "unit": "reqps" }, "overrides": [] }, "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }, "id": 2, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "hideZeros": false, "mode": "single", "sort": "none" } }, "pluginVersion": "12.3.3", "targets": [ { "datasource": { "type": "prometheus", "uid": "prometheus" }, "editorMode": "code", "expr": "sum by (http_route, http_response_status_code) (\r\n rate(http_server_request_duration_seconds_count[2m])\r\n)", "instant": true, "legendFormat": "__auto", "range": true, "refId": "A" } ], "title": "吞吐量", "type": "timeseries" }, { "datasource": { "type": "prometheus", "uid": "prometheus" }, "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisBorderShow": false, "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "barWidthFactor": 0.6, "drawStyle": "line", "fillOpacity": 0, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "showValues": false, "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": 0 }, { "color": "red", "value": 80 } ] }, "unit": "s" }, "overrides": [] }, "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 }, "id": 1, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "hideZeros": false, "mode": "single", "sort": "none" } }, "pluginVersion": "12.3.3", "targets": [ { "datasource": { "type": "prometheus", "uid": "prometheus" }, "editorMode": "code", "expr": "histogram_quantile(0.90, sum by (le, http_route) (rate(http_server_request_duration_seconds_bucket[2m])))\r\n", "instant": true, "legendFormat": "__auto", "range": true, "refId": "A" } ], "title": "延迟时间", "type": "timeseries" }, { "datasource": { "type": "tempo", "uid": "tempo" }, "fieldConfig": { "defaults": { "custom": { "align": "auto", "cellOptions": { "type": "auto" }, "footer": { "reducers": [] }, "inspect": false }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": 0 }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 8, "w": 24, "x": 0, "y": 8 }, "id": 4, "options": { "cellHeight": "sm", "showHeader": true }, "pluginVersion": "12.3.3", "targets": [ { "datasource": { "type": "tempo", "uid": "tempo" }, "filters": [ { "id": "3aaf9885", "operator": "=", "scope": "span" } ], "key": "Q-a3d09f2b-d713-4e41-964b-de64d135e60a-0", "limit": 20, "metricsQueryType": "range", "queryType": "traceqlSearch", "refId": "A", "serviceMapUseNativeHistograms": false, "tableType": "traces" } ], "title": "Traces", "type": "table" } ], "preload": false, "schemaVersion": 42, "tags": [], "templating": { "list": [] }, "time": { "from": "now-1h", "to": "now" }, "timepicker": {}, "timezone": "browser", "title": "OBI dashboard", "uid": "ad8krnh", "version": 10 }

这个Dashboard平凡无奇。但我们可以让它来体现一下Grafana如何展现上文提到的OpenTelemetry的新设计特性Exemplar。当我们在设置里激活Exemplar，我们就可以在曲线图上看到一堆小点，这就是Exemplars，也就是metrics附带的相应的典型的trace/span的索引。点击一个小点，就可以看到详细的信息。点击其中的链接，就可以进入相应的Trace自己的Dashboard。

以下是查看Traces(追踪)的页面：

以下是查看Logs(日志)的页面(没有出错的情况下这个Agent收集的日志较少)：

以下是查看Profiles的页面(我选择了火焰图(Flame Graph)，可以看出不同调用栈的CPU消耗)：

在LGTM的Grafana上，选择Drilldown菜单栏，我们可以看到Grafana自动为Metrics，Traces，Logs，Profiles生成了大量的设计良好的Dashboard Panels。可以选择一些Panel加到自己的Dashboard里。

Grafana在监控系统的界面设计方面确实是独领风骚。但是Grafana LGTM也有一些不足之处。比如上面提到的一个命令就可以安装的Grafana LGTM，这个部署方式对Trace的保留时间只有30分钟，这个对于生产系统肯定是不足的。另外对于较大的系统，或者前端很可能完全不遵循Exemplars的设计模式，Prometheus会难以处理高基数的问题。

4. Signoz

Signoz是另一个极易安装部署的OpenTelemetry后端的集成解决方案。Signoz可以处理OpenTelemetry Metrics，Traces，Logs。Signoz的特点是它采用ClickHouse作为存储数据库，这样就可以大大缓解Metrics的高基数问题。

安装Signoz也非常容易。首先要克隆其Git Repo:

git clone -b main https://github.com/SigNoz/signoz.git

然后到它的目录里启动Docker Compose即可。

cd “signoz的目录”/deploy/docker docker compose up -d --remove-orphans

做Signoz的Dashboard也不难。因为Signoz既支持PromQL，也支持ClickHouse的查询语言(类似SQL)。给Signoz写PromQL有一些小技巧。

因为Signoz直接面对OpenTelemetry的metrics及其label，比如"http.server.request.duration.bucket"及其label(比如http.route)。注意PromQL不允许有“.”用于metric及其label的名称，所以要加引号进行转换。比如下面这PromQL语句：

histogram_quantile(0.90, sum by (le, http_route) (rate(http_server_request_duration_bucket[2m])))

要被改写成：

histogram_quantile(0.90, sum by (le,"http.route") (rate({__name__="http.server.request.duration.bucket"}[2m])))

看起来转换的工作量不大。下面是我做的一个Signoz的Dashboard：

下面是它的源码：

{"description":"","image":"data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMTgiIGhlaWdodD0iMTgiIHZpZXdCb3g9IjAgMCAxOCAxOCIgZmlsbD0ibm9uZSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KPHBhdGggZD0iTTE0Ljk0OTQgMTMuOTQyQzE2LjIzMTggMTIuNDI1OCAxNy4zMjY4IDkuNzAyMiAxNi4xOTU2IDYuNTc0ODdDMTUuNjQ0MyA1LjA1MjQ1IDE1LjAyMTkgNC4yMDI0OSAxNC4yOTY5IDMuNjYyNTJDMTMuODU1NyAzLjMzMzc5IDEyLjA5MzMgMi41MDYzMyA5Ljc1OTY1IDIuODY3NTZDOC4wNTM0OSAzLjEzMjU1IDUuNzc0ODcgNC4yMDg3NCA0LjI5MzY5IDUuOTU5OUMyLjg1NzUyIDcuNjYxMDYgMS43NDg4MyA5LjAwNDc0IDEuNjk3NTggMTAuMzA5N0MxLjYzMTMzIDExLjk4ODMgMi44OTYyNyAxMy40MzA4IDMuMDUwMDEgMTMuNjY0NUMzLjMyMzc0IDE0LjA3OTUgNS4xOTExNSAxNi40NTE4IDguNjk5NzEgMTYuNTczMUMxMS43OTcgMTYuNjc5MyAxMy44MTQ0IDE1LjI4NDQgMTQuOTQ5NCAxMy45NDJaIiBmaWxsPSIjNDAzRDNFIi8+CjxwYXRoIGQ9Ik00LjU1MzYzIDIuNzM3NDdDMi45Mzc0NiAzLjg5MTE2IDEuMTIxMzEgNi4yNTEwMyAxLjQ0NzU0IDkuNTYwODZDMS42MDYyOCAxMS4xNzIgMi4wMDI1MSAxMi4xNDk1IDIuNTcxMjMgMTIuODUwN0MyLjkxNzQ2IDEzLjI3ODIgNC40MTk4OCAxNC41NDkzIDYuNzczNTEgMTQuNzM2OEM5LjE0NTg4IDE0LjkyNTYgMTAuOTQ5NSAxNC4zOTQ0IDEyLjgzMzIgMTMuMDg0NEMxNi42NjE3IDEwLjQyMDggMTYuMDk4IDYuMzkzNTMgMTUuOTM0MyA1LjkyNDhDMTUuNzcwNSA1LjQ1NjA3IDE0LjU0NDQgMi42OTYyMiAxMS4xNzMzIDEuNzE1MDJDOC4xOTg0NCAwLjg1MDA2OCA1Ljk4MzU1IDEuNzE1MDIgNC41NTM2MyAyLjczNzQ3WiIgZmlsbD0iIzVFNjM2NyIvPgo8cGF0aCBkPSJNNy4zOTM1MyAyLjk2MTA5QzUuNjE3MzcgMi44OTczNCAzLjkxOTk2IDQuMjg4NTIgMy43NTYyMiA2LjAwNTkzQzMuNTkyNDggNy43MjIwOSA0LjY1NDkyIDkuMDI5NTIgNi4zMDk4MyA5LjI5NTc2QzcuOTY0NzUgOS41NjA3NCA5Ljg3ODM5IDguNTU1OCAxMC4yNjM0IDYuNDUwOTFDMTAuNjYwOSA0LjI4MjI3IDkuMDg5NjkgMy4wMjIzNCA3LjM5MzUzIDIuOTYxMDlaIiBmaWxsPSJ3aGl0ZSIvPgo8cGF0aCBkPSJNNy45NDIxNyA1LjkwMTE1QzcuOTQyMTcgNS45MDExNSA4LjM2OTY1IDUuODEyNCA4LjQ1NDY1IDUuMTgyNDRDOC41MzgzOSA0LjU2MjQ3IDguMjMwOTEgNC4wMzM3NSA3LjUxMzQ1IDMuODQzNzZDNi43MzM0OSAzLjYzNzUyIDYuMjA0NzcgNC4wNjYyNSA2LjA2NzI3IDQuNTE3NDdDNS44NzYwMyA1LjE0NDk0IDYuMTU4NTIgNS40NDM2NyA2LjE1ODUyIDUuNDQzNjdDNi4xNTg1MiA1LjQ0MzY3IDUuMzkzNTYgNS42Mjc0MSA1LjMzMjMxIDYuNTI5ODdDNS4yNzQ4MSA3LjM4MTA3IDUuODU2MDMgNy44Mzg1NSA2LjQzOTc1IDcuOTc4NTRDNy4xNjA5NiA4LjE1MjI4IDcuOTc4NDIgNy45NTQ3OSA4LjE3ODQxIDcuMDM0ODRDOC4zNDQ2NSA2LjI3NzM4IDcuOTQyMTcgNS45MDExNSA3Ljk0MjE3IDUuOTAxMTVaIiBmaWxsPSIjMzAzMDMwIi8+CjxwYXRoIGQ9Ik02LjczOTgzIDQuNzUzNjJDNi42NzEwOSA1LjAxMjM1IDYuODA4NTggNS4yNjIzNCA3LjA3ODU3IDUuMzMxMDlDNy4zNjk4IDUuNDA0ODMgNy42MzQ3OSA1LjMwODU5IDcuNzA2MDMgNS4wMTExQzcuNzY4NTMgNC43NDczNyA3LjY0MzU0IDQuNTE0ODggNy4zMzYwNSA0LjQzOTg4QzcuMDgzNTcgNC4zNzczOSA2LjgxNDgzIDQuNDcxMTMgNi43Mzk4MyA0Ljc1MzYyWiIgZmlsbD0id2hpdGUiLz4KPHBhdGggZD0iTTYuOTU5NzggNi4wMzk3NEM2LjYzMjMgNS45Mzg0OSA2LjE5OTgyIDYuMDY0NzMgNi4xMzEwNyA2LjUwNDcxQzYuMDYyMzMgNi45NDQ2OSA2LjMyNjA2IDcuMTY5NjggNi42NzEwNCA3LjIzMjE3QzcuMDE2MDMgNy4yOTQ2NyA3LjM0MjI2IDcuMTEzNDMgNy40MDYwMSA2Ljc2MDk1QzcuNDY4NSA2LjQwOTcyIDcuMjg2MDEgNi4xMzk3MyA2Ljk1OTc4IDYuMDM5NzRaIiBmaWxsPSJ3aGl0ZSIvPgo8L3N2Zz4K","layout":[{"h":6,"i":"71ee165f-a071-4849-b238-70321773b564","moved":false,"static":false,"w":6,"x":0,"y":0},{"h":6,"i":"aa82d0dd-a31c-4c86-9000-60318fd040a8","moved":false,"static":false,"w":6,"x":6,"y":0},{"h":8,"i":"aa0d0ebe-72c5-4dbb-854b-1e4ea014cb66","moved":false,"static":false,"w":12,"x":0,"y":6}],"panelMap":{},"tags":[],"title":"MyDash","uploadedGrafana":false,"uuid":"019cc1c5-866f-78ac-8e00-b53cb5194a03","version":"v5","widgets":[{"bucketCount":30,"bucketWidth":0,"columnUnits":{},"contextLinks":{"linksData":[]},"customLegendColors":{},"decimalPrecision":2,"description":"","fillSpans":false,"id":"aa82d0dd-a31c-4c86-9000-60318fd040a8","isLogScale":false,"legendPosition":"bottom","mergeAllActiveQueries":false,"nullZeroValues":"zero","opacity":"1","panelTypes":"graph","query":{"builder":{"queryData":[{"aggregateAttribute":{"dataType":"float64","id":"http.server.request.duration.bucket--float64--Histogram","isColumn":true,"isJSON":false,"key":"http.server.request.duration.bucket","type":"Histogram"},"aggregateOperator":"","aggregations":[{"metricName":"http.server.request.duration.bucket","spaceAggregation":"avg","temporality":"","timeAggregation":"avg"}],"dataSource":"metrics","disabled":false,"expression":"A","filter":{"expression":""},"filters":{"items":[],"op":"AND"},"functions":[],"groupBy":[],"having":{"expression":""},"legend":"","limit":null,"orderBy":[],"queryName":"A","reduceTo":"avg","source":"","spaceAggregation":"","stepInterval":null,"timeAggregation":"rate"}],"queryFormulas":[],"queryTraceOperator":[]},"clickhouse_sql":[{"disabled":false,"legend":"","name":"A","query":"WITH buckets AS (\r\n SELECT\r\n labels['http.route'] AS route,\r\n toFloat64(labels['le']) AS le,\r\n sum(value) AS bucket_value\r\n FROM signoz_metrics.time_series_v2\r\n WHERE metric_name = 'http.server.request.duration.bucket'\r\n AND timestamp >= now() - INTERVAL 2 MINUTE\r\n GROUP BY route, le\r\n),\r\nordered AS (\r\n SELECT\r\n route,\r\n le,\r\n bucket_value,\r\n sum(bucket_value) OVER (PARTITION BY route ORDER BY le ASC) AS cumulative,\r\n sum(bucket_value) OVER (PARTITION BY route) AS total\r\n FROM buckets\r\n)\r\nSELECT\r\n route,\r\n min(le) AS p90\r\nFROM ordered\r\nWHERE cumulative / total >= 0.90\r\nGROUP BY route\r\nORDER BY route"}],"id":"7e7d4900-4f75-46c8-87cf-eff07ea12391","promql":[{"disabled":false,"legend":"","name":"A","query":"histogram_quantile(0.90, sum by (le,\"http.route\") (rate({__name__=\"http.server.request.duration.bucket\"}[2m])))"}],"queryType":"promql","unit":""},"selectedLogFields":[{"dataType":"","fieldContext":"log","fieldDataType":"","isIndexed":false,"name":"timestamp","signal":"logs","type":"log"},{"dataType":"","fieldContext":"log","fieldDataType":"","isIndexed":false,"name":"body","signal":"logs","type":"log"}],"selectedTracesFields":[{"fieldContext":"resource","fieldDataType":"string","name":"service.name","signal":"traces"},{"fieldContext":"span","fieldDataType":"string","name":"name","signal":"traces"},{"fieldContext":"span","fieldDataType":"","name":"duration_nano","signal":"traces"},{"fieldContext":"span","fieldDataType":"","name":"http_method","signal":"traces"},{"fieldContext":"span","fieldDataType":"","name":"response_status_code","signal":"traces"}],"softMax":0,"softMin":0,"stackedBarChart":false,"thresholds":[],"timePreferance":"GLOBAL_TIME","title":"延迟时间","yAxisUnit":"s"},{"bucketCount":30,"bucketWidth":0,"columnUnits":{},"contextLinks":{"linksData":[]},"customLegendColors":{},"decimalPrecision":2,"description":"","fillSpans":false,"id":"71ee165f-a071-4849-b238-70321773b564","isLogScale":false,"legendPosition":"bottom","mergeAllActiveQueries":false,"nullZeroValues":"zero","opacity":"1","panelTypes":"graph","query":{"builder":{"queryData":[{"aggregateAttribute":{"dataType":"float64","id":"http.server.request.duration.count--float64--Sum","isColumn":true,"isJSON":false,"key":"http.server.request.duration.count","type":"Sum"},"aggregateOperator":"","aggregations":[{"metricName":"http.server.request.duration.count","reduceTo":"sum","spaceAggregation":"sum","temporality":"","timeAggregation":"rate"}],"dataSource":"metrics","disabled":false,"expression":"A","filter":{"expression":""},"filters":{"items":[],"op":"AND"},"functions":[],"groupBy":[],"having":{"expression":""},"legend":"","limit":null,"orderBy":[],"queryName":"A","reduceTo":"avg","source":"","spaceAggregation":"","stepInterval":null,"timeAggregation":"rate"}],"queryFormulas":[],"queryTraceOperator":[]},"clickhouse_sql":[{"disabled":false,"legend":"","name":"A","query":""}],"id":"b601e954-67d7-40a0-8aa3-c8c2e4c6a371","promql":[{"disabled":false,"legend":"","name":"A","query":"sum by (\"http.route\", \"http.response.status_code\") (rate({__name__=\"http.server.request.duration.count\"}[2m]))"}],"queryType":"promql","unit":""},"selectedLogFields":[{"dataType":"","fieldContext":"log","fieldDataType":"","isIndexed":false,"name":"timestamp","signal":"logs","type":"log"},{"dataType":"","fieldContext":"log","fieldDataType":"","isIndexed":false,"name":"body","signal":"logs","type":"log"}],"selectedTracesFields":[{"fieldContext":"resource","fieldDataType":"string","name":"service.name","signal":"traces"},{"fieldContext":"span","fieldDataType":"string","name":"name","signal":"traces"},{"fieldContext":"span","fieldDataType":"","name":"duration_nano","signal":"traces"},{"fieldContext":"span","fieldDataType":"","name":"http_method","signal":"traces"},{"fieldContext":"span","fieldDataType":"","name":"response_status_code","signal":"traces"}],"softMax":0,"softMin":0,"stackedBarChart":false,"thresholds":[],"timePreferance":"GLOBAL_TIME","title":"吞吐量","yAxisUnit":"{req}/s"},{"bucketCount":30,"bucketWidth":0,"columnUnits":{},"columnWidths":{"date":145,"duration_nano":145,"http_method":145,"name":145,"response_status_code":145,"service.name":145},"contextLinks":{"linksData":[]},"customLegendColors":{},"decimalPrecision":2,"description":"","fillSpans":false,"id":"aa0d0ebe-72c5-4dbb-854b-1e4ea014cb66","isLogScale":false,"legendPosition":"bottom","mergeAllActiveQueries":false,"nullZeroValues":"zero","opacity":"1","panelTypes":"list","query":{"builder":{"queryData":[{"aggregations":[{"expression":"count() "}],"dataSource":"traces","disabled":false,"expression":"A","filter":{"expression":""},"functions":[],"groupBy":[],"having":{"expression":""},"legend":"","limit":null,"orderBy":[],"queryName":"A","selectColumns":[{"fieldContext":"resource","fieldDataType":"string","name":"service.name","signal":"traces"},{"fieldContext":"span","fieldDataType":"string","name":"name","signal":"traces"},{"fieldContext":"span","fieldDataType":"","name":"duration_nano","signal":"traces"},{"fieldContext":"span","fieldDataType":"","name":"http_method","signal":"traces"},{"fieldContext":"span","fieldDataType":"","name":"response_status_code","signal":"traces"}],"source":"","stepInterval":null}],"queryFormulas":[],"queryTraceOperator":[]},"clickhouse_sql":[{"disabled":false,"legend":"","name":"A","query":""}],"id":"8f01907d-ee09-4031-b0ac-c0307d36c652","promql":[{"disabled":false,"legend":"","name":"A","query":""}],"queryType":"builder","unit":""},"selectedLogFields":[{"dataType":"","fieldContext":"log","fieldDataType":"","isIndexed":false,"name":"timestamp","signal":"logs","type":"log"},{"dataType":"","fieldContext":"log","fieldDataType":"","isIndexed":false,"name":"body","signal":"logs","type":"log"}],"selectedTracesFields":[{"fieldContext":"resource","fieldDataType":"string","name":"service.name","signal":"traces"},{"fieldContext":"span","fieldDataType":"string","name":"name","signal":"traces"},{"fieldContext":"span","fieldDataType":"","name":"duration_nano","signal":"traces"},{"fieldContext":"span","fieldDataType":"","name":"http_method","signal":"traces"},{"fieldContext":"span","fieldDataType":"","name":"response_status_code","signal":"traces"}],"softMax":0,"softMin":0,"stackedBarChart":false,"thresholds":[],"timePreferance":"GLOBAL_TIME","title":"","yAxisUnit":"none"}]}

到目前为止，似乎Signoz的Dashboard还不支持Exemplar。

下面是Signoz自带的Trace的Dashboard：

Signoz的界面虽然不如Grafana华丽，但是其实用性很强。如果用户选择全开源的系统，Signoz应该也是一个重要的选项。

5. 总结

OpenTelemetry(OTel)作为应用系统监控的开放标准，通过解耦前端和后端系统实现了灵活切换监控平台的能力。文章介绍了支持OTel的开源后端系统解决方案：1)Prometheus+Jaeger+Elasticsearch组合，分别处理指标、追踪和日志，但存在集成度不高的问题；2)Grafana LGTM一体化方案(Loki+Grafana+Tempo+Mimir)，提供统一界面和Exemplar特性支持；3)Signoz基于ClickHouse的解决方案，有效缓解高基数问题。这些系统各具优势，Grafana在界面设计上领先，Signoz在数据处理上更高效，而传统组合则组件成熟但集成度较低。