{"id":1054,"date":"2021-04-25T15:36:35","date_gmt":"2021-04-25T07:36:35","guid":{"rendered":"https:\/\/www.linuxdevops.cn\/?p=1054"},"modified":"2023-04-25T10:26:37","modified_gmt":"2023-04-25T02:26:37","slug":"prometheus-operator-configures-alarm-rules-and-adds-pin","status":"publish","type":"post","link":"https:\/\/www.linuxdevops.cn\/2021\/04\/prometheus-operator-configures-alarm-rules-and-adds-pin\/","title":{"rendered":"Prometheus Operator \u81ea\u5b9a\u4e49\u62a5\u8b66\u89c4\u5219\u548c\u6dfb\u52a0\u9489\u9489\u62a5\u8b66"},"content":{"rendered":"
\n\u901a\u8fc7helm \u5b89\u88c5\u90e8\u7f72 \u5b98\u65b9\u6587\u6863<\/strong><\/a> \u6d4b\u8bd5\u5b89\u88c5\u8fd8\u662f\u5f88\u7b80\u5355\u7684\uff0c\u5b98\u65b9\u6587\u6863\u66f4\u52a0\u8be6\u7ec6\uff0c\u672c\u6587\u4e3b\u8981\u505aPrometheus Operator \u914d\u7f6e\u62a5\u8b66\u89c4\u5219\u548c\u6dfb\u52a0\u9489\u9489\u62a5\u8b66<\/p>\n<\/blockquote>\n
\u81ea\u5b9a\u4e49\u62a5\u8b66\u89c4\u5219<\/h2>\n
\u6211\u4eec\u5b89\u88c5\u597d Prometheus Operator \u4e4b\u540e\uff0c\u6253\u5f00 Prometheus Dashboard \u9875\u9762
Alerts<\/code>\u9875\u9762\u80fd\u770b\u5230\u597d\u591a\u62a5\u8b66\u89c4\u5219\u3002<\/p>\n
<\/p>\n
\u8fd9\u4e9b\u544a\u8b66\u89c4\u5219\u6765\u81ea\u4e8ePrometheus \u7684\u914d\u7f6e\u6587\u4ef6\u4e4b\u4e2d\u6307\u5b9a AlertManager \u5b9e\u4f8b\u548c \u62a5\u8b66\u7684 rules \u6587\u4ef6\uff0c\u6211\u4eec\u53ef\u4ee5\u5728Prometheus Dashboard \u7684Config \u9875\u9762\u4e0b\u9762\u67e5\u770b\u5173\u4e8e AlertManager \u7684\u914d\u7f6e\uff1a<\/p>\n
<\/p>\n
\u4e0a\u9762 alertmanagers \u5b9e\u4f8b\u7684\u914d\u7f6e\u6211\u4eec\u53ef\u4ee5\u770b\u5230\u662f\u901a\u8fc7\u89d2\u8272\u4e3a endpoints \u7684 kubernetes \u7684\u670d\u52a1\u53d1\u73b0\u673a\u5236\u83b7\u53d6\u7684\uff0c\u5339\u914d\u7684\u662f\u5728monitor \u540d\u79f0\u7a7a\u95f4\u4e0bservice_nam\u4e3a prometheus-kube-prometheus-alertmanager\uff0cendpoint_port_name\u4e3a web \u7684 Service \u670d\u52a1\uff0c\u6211\u4eec\u67e5\u770b\u4e0b prometheus-kube-prometheus-alertmanager \u8fd9\u4e2a Service\uff1a<\/p>\n
<\/p>\n
\u53ef\u4ee5\u770b\u5230\u670d\u52a1\u540d\u6b63\u662f prometheus-kube-prometheus-alertmanager\uff0cPort \u5b9a\u4e49\u7684\u540d\u79f0\u4e5f\u662f web\uff0c\u7b26\u5408\u4e0a\u9762\u7684\u89c4\u5219\uff0c
\u6240\u4ee5 Prometheus \u548c AlertManager \u7ec4\u4ef6\u5c31\u6b63\u786e\u5173\u8054\u4e0a\u4e86\u3002<\/code>\u800c\u5bf9\u5e94\u7684\u62a5\u8b66\u89c4\u5219\u6587\u4ef6\u4f4d\u4e8e\uff1a
\/etc\/prometheus\/rules\/prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0\/*.yaml<\/code>\u76ee\u5f55\u4e0b\u9762\u6240\u6709\u7684 YAML \u6587\u4ef6\u3002\u6211\u4eec\u53ef\u4ee5\u8fdb\u5165 Prometheus \u7684 Pod \u4e2d\u9a8c\u8bc1\u4e0b\u8be5\u76ee\u5f55\u4e0b\u9762\u662f\u5426\u6709 YAML \u6587\u4ef6\uff1a<\/p>\n
<\/p>\n
\u8fd9\u4e9b\u6587\u4ef6\u5c31\u662f\u6839\u636e PrometheusRule K8s\u81ea\u5b9a\u4e49\u8d44\u6e90\u6587\u4ef6\u521b\u5efa\u7684Prometheus\u62a5\u8b66\u89c4\u5219\u3002\u6240\u4ee5\u6211\u4eec\u9700\u8981\u4fee\u6539\u6216\u8005\u81ea\u5b9a\u4e49\u6dfb\u52a0\u4e00\u4e2a\u62a5\u8b66\u89c4\u5219\u7684\u8bdd\uff0c\u53ea\u9700\u8981\u521b\u5efa\u4e00\u4e2aPrometheusRule \u8d44\u6e90\u5bf9\u8c61\u5373\u53ef\u3002Prometheus\u4f1a\u6839\u636e\u521b\u5efa\u7684PrometheusRule \u8d44\u6e90\u81ea\u52a8\u5728\u8fd9\u4e2a\u76ee\u5f55\u4e0b\u751f\u6210 .yaml \u6587\u4ef6\u3002<\/p>\n
\u4e3a\u4ec0\u4e48 Prometheus \u80fd\u591f\u8bc6\u522b\u8fd9\u4e2a PrometheusRule \u8d44\u6e90\u5bf9\u8c61\u5462\uff1f\u662f\u56e0\u4e3a\u6211\u4eec\u7684Prometheus \u914d\u7f6e\u6587\u4ef6\u91cc\u9762\u6709\u975e\u5e38\u91cd\u8981\u7684\u4e00\u4e2a\u5c5e\u6027 ruleSelector\uff0c\u662f\u7528\u6765\u5339\u914d rule \u89c4\u5219\u7684\u8fc7\u6ee4\u5668\uff0c\u53ea\u8981PrometheusRule \u8d44\u6e90 \u5e26\u6709app: kube-prometheus-stack \u548c release: prometheus \u8fd9\u4e24\u4e2a\u6807\u7b7e\uff0cPrometheus \u5c31\u80fd\u591f\u5339\u914d\u5230\u4e86\u3002<\/p>\n
\nPrometheus \u7684\u914d\u7f6e\u6587\u4ef6\u6765\u6e90\u4e8e prometheus-prometheus-kube-prometheus-prometheus Secret\u8d44\u6e90\u5bf9\u8c61\u3002\u4e0d\u660e\u767d\u7684\u53ef\u4ee5\u67e5\u770bPrometheus Operator\u7ec4\u4ef6\u7b80\u4ecb\u914d\u7f6e\u89e3\u6790<\/a><\/p>\n<\/blockquote>\n
<\/p>\n
\u6dfb\u52a0\u4e00\u4e2a\u81ea\u5b9a\u4e49\u62a5\u8b66\u89c4\u5219,\u96c6\u7fa4\u8282\u70b9\u78c1\u76d8\u4f7f\u7528\u7387\u64cd\u8fc7 80% \u5c31\u62a5\u8b66\u3002\u6ce8\u610f\u5fc5\u987b\u5e26\u6709app: kube-prometheus-stack \u548c release: prometheus \u8fd9\u4e24\u4e2a\u6807\u7b7e\u3002<\/p>\n
apiVersion: monitoring.coreos.com\/v1\nkind: PrometheusRule\nmetadata:\n labels:\n app: kube-prometheus-stack\n release: prometheus\n name: disk-free-rules\n namespace: monitor\nspec:\n groups:\n - name: disk\n rules:\n - alert: diskFree\n annotations:\n summary: "{{ $labels.job }} \u9879\u76ee\u5b9e\u4f8b {{ $labels.instance }} \u78c1\u76d8\u4f7f\u7528\u7387\u5927\u4e8e 80%"\n description: "{{ $labels.instance }} {{ $labels.mountpoint }} \u78c1\u76d8\u4f7f\u7528\u7387\u5927\u4e8e80% (\u5f53\u524d\u7684\u503c: {{ $value }}%),\u8bf7\u53ca\u65f6\u5904\u7406"\n expr: |\n (1-(node_filesystem_free_bytes{fstype=~"ext4|xfs",mountpoint!="\/boot"} \/ node_filesystem_size_bytes{fstype=~"ext4|xfs",mountpoint!="\/boot"}) )*100 > 80\n for: 1m\n labels:\n level: disaster\n severity: warning<\/code><\/pre>\n
\u521b\u5efa\u597d\u540e\u5c31\u53ef\u4ee5\u5728 Prometheus Dashboard \u7684 Alert \u9875\u9762\u4e0b\u9762\u770b\u5230\u65b0\u5efa\u7684\u62a5\u8b66\u89c4\u5219\u4e86;\u540c\u65f6\u4e5f\u53ef\u4ee5\u5728Prometheus pod \u91cc\u7684
\/etc\/prometheus\/rules\/prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0\/ <\/code> \u76ee\u5f55\u770b\u5230\u65b0\u5efa\u7684YAML \u6587\u4ef6\u3002<\/p>\n
<\/p>\n
<\/p>\n
\u914d\u7f6e\u9489\u9489\u62a5\u8b66<\/h2>\n
\u6211\u4eec\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539Alermanager \u7684\u914d\u7f6e\u6587\u4ef6\u6765\u914d\u7f6e\u5404\u79cd\u62a5\u8b66\u63a5\u6536\u5668\u3002<\/p>\n
\nAlermanager \u7684\u914d\u7f6e\u6587\u4ef6\u6765\u6e90\u4e8e alertmanager-prometheus-kube-prometheus-alertmanager Secret\u8d44\u6e90\u5bf9\u8c61\u3002\u4e0d\u660e\u767d\u7684\u53ef\u4ee5\u67e5\u770bPrometheus Operator\u7ec4\u4ef6\u7b80\u4ecb\u914d\u7f6e\u89e3\u6790<\/a><\/p>\n<\/blockquote>\n
\u6211\u4eec\u53ef\u4ee5\u5148\u5728Alertmanager Dashboard \u67e5\u770b\u4e0b\u9ed8\u8ba4\u7684Config \u914d\u7f6e\u3002<\/p>\n
<\/p>\n
Alermanager \u7684\u914d\u7f6e\u6587\u4ef6\u6765\u6e90\u4e8e alertmanager-prometheus-kube-prometheus-alertmanager Secret\u8d44\u6e90\u5bf9\u8c61<\/p>\n
<\/p>\n
\u6211\u4eec\u5bf9alertmanager.yaml \u5185\u5bb9\u89e3\u6790\uff0c\u67e5\u770b\u4e0b\u5185\u5bb9\u3002
k9s \u8fd9\u4e2a\u5de5\u5177\u53ef\u4ee5\u76f4\u63a5\u5bf9secret\u5185\u5bb9\u8fdb\u884c\u89e3\u6790\uff0c\u6ca1\u6709\u8fd9\u4e2a\u5de5\u5177\u7684\u4e5f\u53ef\u4ee5\u4f7f\u7528 base64 -d \u5bf9\u5185\u5bb9\u8fdb\u884c\u89e3\u6790\u3002<\/code><\/p>\n
<\/p>\n
\u53ef\u4ee5\u770b\u770b\u5185\u5bb9\u548c Alertmanager Dashboard \u770b\u5230\u7684Config\u914d\u7f6e\u662f\u4e00\u6837\u7684\uff0c\u6211\u4eec\u76f4\u63a5\u4fee\u6539\u8fd9\u4e2aSecret \u5c31\u53ef\u4ee5\u4e86\u3002<\/p>\n
global:\n resolve_timeout: 5m\n receivers:\n - name: dingtalk-webhook\n webhook_configs:\n - send_resolved: true\n url: "http:\/\/alertmanager-webhook-dingtalk.monitor\/dingtalk\/webhook1\/send"\n route:\n group_by:\n - job\n group_interval: 5m\n group_wait: 30s\n receiver: dingtalk-webhook\n repeat_interval: 12h\n routes:\n - receiver: dingtalk-webhook\n group_wait: 10s\n templates:\n - '\/etc\/alertmanager\/config\/*.tmpl'<\/code><\/pre>\n
\u5c06\u4e0a\u9762\u6587\u4ef6\u4fdd\u5b58\u4e3a alertmanager-prometheus-kube-prometheus-alertmanager.yaml\uff0c\u7136\u540e\u4f7f\u7528\u8fd9\u4e2a\u6587\u4ef6\u521b\u5efa\u4e00\u4e2a Secret \u5bf9\u8c61\uff1a<\/p>\n
#\u5148\u5220\u9664\u4e4b\u524d\u7684secret \u5bf9\u8c61\n$ kubectl delete alertmanager-prometheus-kube-prometheus-alertmanager -n monitor \n# \u521b\u5efa\u65b0\u7684secret\u5bf9\u8c61\n$ kubectl create secret genericalertmanager-prometheus-kube-prometheus-alertmanager --from-file=alertmanager-prometheus-kube-prometheus-alertmanager -n monitoring<\/code><\/pre>\n
\u8fd8\u6709\u4e00\u79cd\u65b9\u5f0f\uff0c\u5c31\u662f\u6211\u662f\u901a\u8fc7helm \u90e8\u7f72\u7684\uff0c\u6240\u4ee5\u6211\u76f4\u63a5\u4fee\u6539\u7684helm values.yaml \u6587\u4ef6\uff0c\u7136\u540e\u66f4\u65b0\uff0chelm \u81ea\u52a8\u6e32\u67d3\u65b0\u7684\u914d\u7f6e\u3002<\/p>\n
<\/p>\n
\u4fee\u6539\u540e\u66f4\u65b0\uff1b<\/p>\n
$ helm upgrade prometheus -n monitor .<\/code><\/pre>\n
\u521b\u5efa\u9489\u9489\u544a\u8b66\u6a21\u677f\u548c\u914d\u7f6e\u6587\u4ef6;<\/p>\n
## vim dingTalk-webhook-configmap.yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n namespace: monitor\n name: dingtalk-webhook-config\ndata:\n config.yaml: |\n ## Request timeout\n timeout: 5s\n\n ## Customizable templates path\n templates:\n - \/config\/template.tmpl\n\n ## You can also override default template using `default_message`\n ## The following example to use the 'legacy' template from v0.3.0\n # default_message:\n # title: '{{ template "legacy.title" . }}'\n # text: '{{ template "legacy.content" . }}'\n targets:\n webhook1:\n url: https:\/\/oapi.dingtalk.com\/robot\/send?access_token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n # secret for signature\n # secret: SEC000000000000000000000\n message:\n title: '{{ template "ding.link.title" . }}'\n text: '{{ template "ding.link.content" . }}'\n mention:\n all: true\n mention:\n mobiles: ['150xxxxxxxx']\n template.tmpl: |\n {{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}\n {{ define "__alertmanagerURL" }}{{ .ExternalURL }}\/#\/alerts?receiver={{ .Receiver }}{{ end }}\n\n {{ define "__text_alert_list" }}{{ range . }}\n **Labels**\n {{ range .Labels.SortedPairs }}> - {{ .Name }}: {{ .Value | markdown | html }}\n {{ end }}\n **Annotations**\n {{ range .Annotations.SortedPairs }}> - {{ .Name }}: {{ .Value | markdown | html }}\n {{ end }}\n **Source:** [{{ .GeneratorURL }}]({{ .GeneratorURL }})\n {{ end }}{{ end }}\n\n {{ define "default.__text_alert_list" }}{{ range . }}\n ---\n **\u544a\u8b66\u7ea7\u522b:** {{ .Labels.severity | upper }}\n\n **\u8fd0\u8425\u56e2\u961f:** {{ .Labels.team | upper }}\n\n **\u89e6\u53d1\u65f6\u95f4:** {{ dateInZone "2006.01.02 15:04:05" (.StartsAt) "Asia\/Shanghai" }}\n\n **\u4e8b\u4ef6\u4fe1\u606f:**\n {{ range .Annotations.SortedPairs }}> - {{ .Name }}: {{ .Value | markdown | html }}\n\n {{ end }}\n\n **\u4e8b\u4ef6\u6807\u7b7e:**\n {{ range .Labels.SortedPairs }}{{ if and (ne (.Name) "severity") (ne (.Name) "summary") (ne (.Name) "team") }}> - {{ .Name }}: {{ .Value | markdown | html }}\n {{ end }}{{ end }}\n {{ end }}\n {{ end }}\n {{ define "default.__text_alertresovle_list" }}{{ range . }}\n ---\n **\u544a\u8b66\u7ea7\u522b:** {{ .Labels.severity | upper }}\n\n **\u8fd0\u8425\u56e2\u961f:** {{ .Labels.team | upper }}\n\n **\u89e6\u53d1\u65f6\u95f4:** {{ dateInZone "2006.01.02 15:04:05" (.StartsAt) "Asia\/Shanghai" }}\n\n **\u7ed3\u675f\u65f6\u95f4:** {{ dateInZone "2006.01.02 15:04:05" (.EndsAt) "Asia\/Shanghai" }}\n\n **\u4e8b\u4ef6\u4fe1\u606f:**\n {{ range .Annotations.SortedPairs }}> - {{ .Name }}: {{ .Value | markdown | html }}\n\n {{ end }}\n\n **\u4e8b\u4ef6\u6807\u7b7e:**\n {{ range .Labels.SortedPairs }}{{ if and (ne (.Name) "severity") (ne (.Name) "summary") (ne (.Name) "team") }}> - {{ .Name }}: {{ .Value | markdown | html }}\n {{ end }}{{ end }}\n {{ end }}\n {{ end }}\n\n {{\/* Default *\/}}\n {{ define "default.title" }}{{ template "__subject" . }}{{ end }}\n {{ define "default.content" }}#### \\[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}\\] **[{{ index .GroupLabels "alertname" }}]({{ template "__alertmanagerURL" . }})**\n {{ if gt (len .Alerts.Firing) 0 -}}\n\n ![\u8b66\u62a5 \u56fe\u6807](https:\/\/www.wangfeng.live\/wp-content\/uploads\/2021\/04\/4e69b9298eccf7966aebb996708a7f12.jpg)\n **====\u4fa6\u6d4b\u5230\u6545\u969c====**\n {{ template "default.__text_alert_list" .Alerts.Firing }}\n\n {{- end }}\n\n {{ if gt (len .Alerts.Resolved) 0 -}}\n {{ template "default.__text_alertresovle_list" .Alerts.Resolved }}\n\n {{- end }}\n {{- end }}\n\n {{\/* Legacy *\/}}\n {{ define "legacy.title" }}{{ template "__subject" . }}{{ end }}\n {{ define "legacy.content" }}#### \\[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}\\] **[{{ index .GroupLabels "alertname" }}]({{ template "__alertmanagerURL" . }})**\n {{ template "__text_alert_list" .Alerts.Firing }}\n {{- end }}\n\n {{\/* Following names for compatibility *\/}}\n {{ define "ding.link.title" }}{{ template "default.title" . }}{{ end }}\n {{ define "ding.link.content" }}{{ template "default.content" . }}{{ end }}<\/code><\/pre>\n
\u521b\u5efa\u9489\u9489\u544a\u8b66\u8d44\u6e90\u6e05\u5355\uff1b<\/p>\n
## vim alertmanager-webhook-dingtalk.yaml\n\napiVersion: apps\/v1\nkind: Deployment\nmetadata:\n namespace: monitor\n name: alertmanager-webhook-dingtalk\n labels:\n app: alertmanager-webhook-dingtalk\nspec:\n selector:\n matchLabels:\n app: alertmanager-webhook-dingtalk\n replicas: 1\n template:\n metadata:\n labels:\n app: alertmanager-webhook-dingtalk\n spec:\n volumes:\n - name: dingtalk-webhook-confing\n configMap:\n name: dingtalk-webhook-config\n containers:\n - name: alertmanager-webhook-dingtalk\n image: timonwong\/prometheus-webhook-dingtalk\n args:\n - --web.listen-address=:8060\n - --config.file=\/config\/config.yaml\n volumeMounts:\n - name: dingtalk-webhook-confing\n mountPath: \/config\n resources:\n limits:\n cpu: 100m\n memory: 100Mi\n ports:\n - name: http\n containerPort: 8060\n---\napiVersion: v1\nkind: Service\nmetadata:\n namespace: monitor\n name: alertmanager-webhook-dingtalk\n labels:\n app: alertmanager-webhook-dingtalk\nspec:\n selector:\n app: alertmanager-webhook-dingtalk\n ports:\n - name: http\n port: 80\n targetPort: http<\/code><\/pre>\n
\u7136\u540e\u521b\u5efa\u8d44\u6e90\uff1b<\/p>\n
$ kubectl apply -f dingTalk-webhook-configmap.yaml\n $ kubectl apply -f alertmanager-webhook-dingtalk.yaml<\/code><\/pre>\n
\u7136\u540e\u6d4b\u8bd5\uff0c\u6211\u4eec\u628a\u521a\u81ea\u5b9a\u4e49\u521b\u5efa\u7684\u62a5\u8b66\u89c4\u5219\u6539\u4e0b\uff0c\u8be5\u4e3a\u78c1\u76d8\u4f7f\u7528\u7387\u5927\u4e8e 10% \u5c31\u544a\u8b66\u3002<\/p>\n
apiVersion: monitoring.coreos.com\/v1\nkind: PrometheusRule\nmetadata:\n labels:\n app: kube-prometheus-stack\n release: prometheus\n name: disk-free-rules\n namespace: monitor\nspec:\n groups:\n - name: disk\n rules:\n - alert: diskFree\n annotations:\n summary: "{{ $labels.job }} \u9879\u76ee\u5b9e\u4f8b {{ $labels.instance }} \u78c1\u76d8\u4f7f\u7528\u7387\u5927\u4e8e 80%"\n description: "{{ $labels.instance }} {{ $labels.mountpoint }} \u78c1\u76d8\u4f7f\u7528\u7387\u5927\u4e8e80% (\u5f53\u524d\u7684\u503c: {{ $value }}%),\u8bf7\u53ca\u65f6\u5904\u7406"\n expr: |\n (1-(node_filesystem_free_bytes{fstype=~"ext4|xfs",mountpoint!="\/boot"} \/ node_filesystem_size_bytes{fstype=~"ext4|xfs",mountpoint!="\/boot"}) )*100 > 10\n for: 1m\n labels:\n level: disaster\n severity: warning<\/code><\/pre>\n
#\u66f4\u65b0\u4e0b\n $ kubectl apply -f disk-free-rules.yaml<\/code><\/pre>\n
\u67e5\u770bPrometheus Dashboard Alerts \u9875\u9762\u5df2\u7ecf\u51fa\u53d1\u62a5\u8b66\u4e86\u3002<\/p>\n
<\/p>\n
\u67e5\u770b\u9489\u9489\u5df2\u7ecf\u6536\u5230\u544a\u8b66\u4fe1\u606f\u4e86\u3002
\u63cf\u8ff0\u4fe1\u606f\u6ca1\u6539\u6240\u4ee5\u663e\u793a80%\uff0c\u5c31\u8fd9\u6837\u5427\u5b9e\u9a8c\u5df2\u7ecf\u6210\u529f\u4e86\u3002<\/code><\/p>\n
<\/p>\n
\n\u53c2\u8003\uff1a<\/p>\n
Helm kube-prometheus-stack<\/strong><\/a><\/p>\n
prometheus operator \u7ec4\u4ef6\u7b80\u4ecb<\/strong><\/a><\/p>\n