
OpenObserve是一个 Rust 开发的开源的高性能云原生可观测平台(日志、指标、追踪),比起 ElasticSearch 它大约可以节省 140 倍的存储成本,OpenObserve 能够处理 PB 级的数据,如果你正在寻找一个用于日志、指标、追踪的可观测工具,那么 OpenObserve 是非常值得尝试的。OpenObserve 虽然目前处于 alpha 阶段,但其实也进行了广泛的测试。
Elasticsearch 是一个通用搜索引擎,可以使用应用程序搜索或日志搜索。OpenObserve 是专门为日志搜索而构建的,如果你正在寻找 Elasticsearch 的轻量级替代品,那么您应该看看 ZincSearch,如果只是想要一个日志搜索引擎,那么 OpenObserve 是一个非常好的选择。
OpenObserve 不依赖于数据索引,它将未索引的数据以压缩格式存储在本地磁盘或以 parquet 列格式的对象存储中。这使得数据摄取期间的计算要求大大降低,并且压缩率非常高,从而使存储成本降低约 140 倍。没有数据索引意味着全扫描搜索可能比 Elasticsearch 慢,但由于分区和缓存等多种其他技术,仍然应该很快。Uber 发现其生产环境中 80% 的查询是聚合查询,而 OpenObserve 的列式数据存储意味着聚合查询通常比 Elasticsearch 快得多。

下面是我们使用 Fluentbit 将真实日志数据从 Kubernetes 集群发送到 Elasticsearch 和 OpenObserve 时的结果,这只与存储有关。EBS 卷的成本为 8 美分/GB/月 (GP3),s3 的成本为 2.3 美分/GB/月。在 Elasticsearch 的 HA 模式下,通常有 1 个主节点和 2 个副本。无需复制 s3 来实现数据持久性/可用性,因为 AWS 会将你的对象冗余存储在 Amazon S3 区域中至少三个可用区 (AZ) 的多个设备上。
OpenObserve VS Elasticsearch
在上述场景中,OpenObserve 具有比 Elasticsearch 低 140 倍的存储成本的显著优势,这甚至没有考虑额外未使用的 EBS 卷容量(为了不耗尽磁盘空间而需要提供这些容量)以及持续监控磁盘使用情况以使其不被填满所需的工作。
无状态节点架构允许 OpenObserve 水平扩展,而无需担心数据复制或损坏。与 Elasticsearch 相比,您通常会发现管理 OpenObserve 集群的运维工作量和成本要低得多。
OpenObserve 内置的图形用户界面消除了对 Kibana 等其他组件的需求,而且由于 Rust 的优势,性能出色,而无需面对 JVM 所带来的问题。
与 Elasticsearch 相比,Elasticsearch 是一个通用性的搜索引擎,同时也兼具观测工具的功能。而 OpenObserve 是从头开始构建的观测工具,非常注重提供优秀的可观测性能。
OpenObserve 可以在单节点下运行,也可以在集群中以 HA 模式运行。
单节点模式也分几种架构,主要是数据存储的方式不同,主要有如下几种:
Sled 和本地磁盘模式
如果你只需要进行简单使用和测试,或者对高可用性没有要求,可以使用此模式。当然你仍然可以在一台机器上每天处理超过 2 TB 的数据。在我们的测试中,使用默认配置,Mac M2 的处理速度为约 31 MB/秒,即每分钟处理 1.8 GB,每天处理 2.6 TB。该模式也是运行 OpenObserve 的默认模式。
Sled本地模式
Sled 和对象存储模式
该模式和 OpenObserve 的默认模式基本上一致,只是数据存在了对象存储中,这样可以更好的支持高可用性,因为数据不会丢失。
Sled对象存储模式
Etcd 和对象存储模式
该模式是使用 Etcd 来存储元数据,数据仍然存储在对象存储中。
Etcd对象存储模式
HA 模式不支持本地磁盘存储,集群模式下 OpenObserve 会运行多个节点,每个节点都是无状态的,数据存储在对象存储中,元数据存储在 Etcd 中,这样可以更好的支持高可用性,因为数据不会丢失。
Etcd对象存储
在该模式下 OpenObserve 主要包括 Router、Querier、Ingester 和 Compactor 四个组件,这些组件都可以水平扩展;Etcd 用于存储用户、函数、报警规则和集群节点信息等元数据;对象存储(例如 s3、minio、gcs 等等)存储parquet文件和文件列表索引的所有数据。
OpenObserve 的安装非常简单,只需要下载二进制文件即可,它支持 linux、Windows 和 MacOS,也支持 Docker 镜像。我们这里当然还是将其安装到 Kubernetes 集群中,为简单这里我们直接使用默认的 Sled 和本地磁盘模式。
首先创建一个命名空间:
$ kubectl create ns openobserve
然后创建如下所示的资源清单文件:
# openobserve.yamlapiVersion: v1kind: Servicemetadata:name: openobservenamespace: openobservespec:clusterIP: Noneselector:app: openobserveports:- name: httpport: 5080targetPort: 5080---# create statefulsetapiVersion: apps/v1kind: StatefulSetmetadata:name: openobservenamespace: openobservelabels:app: openobservespec:serviceName: openobservereplicas: 1selector:matchLabels:app: openobservetemplate:metadata:labels:app: openobservespec:securityContext:fsGroup: 2000runAsUser: 10000runAsGroup: 3000runAsNonRoot: truecontainers:- name: openobserveimage: public.ecr.aws/zinclabs/openobserve:latestenv:- name: ZO_ROOT_USER_EMAIL # 指定管理员邮箱value: [emailprotected]- name: ZO_ROOT_USER_PASSWORD # 指定管理员密码value: root321- name: ZO_DATA_DIRvalue: /dataimagePullPolicy: Alwaysresources:limits:cpu: 4096mmemory: 2048Mirequests:cpu: 256mmemory: 50Miports:- containerPort: 5080name: httpvolumeMounts:- name:>
创建后我们可以查看一下 OpenObserve 的日志来验证是否启动成功:
$ kubectl logs -f openobserve-0 -n openobserve[2023-08-04T10:18:06Z INFOopenobserve] Starting OpenObserve v0.5.1[2023-08-04T10:18:06Z INFOopenobserve::service::db::user] get; org_id=Some("default")/cdn-cgi/l/email-protection">[emailprotected]"[2023-08-04T10:18:06Z INFOtracing::span] set;[2023-08-04T10:18:06Z INFOopenobserve::service::db::user] Users Cached# ......[2023-08-04T10:18:06Z INFOopenobserve::common::meta::telemetry] sending event OpenObserve - Starting server[2023-08-04T10:18:07Z INFOactix_server::builder] starting 4 workers[2023-08-04T10:18:07Z INFOactix_server::server] Tokio runtime found; starting in existing Tokio runtime[2023-08-04T10:18:07Z INFOopenobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0[2023-08-04T10:18:07Z INFOopenobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0[2023-08-04T10:18:07Z INFOopenobserve] starting HTTP server at: 0.0.0.0:5080, thread_id: 0
启动后我们可以通过kubectl port-forward命令将 OpenObserve 的 5080 端口映射到本地,然后在浏览器中访问即可看到 OpenObserve 的 UI 界面。
$ kubectl port-forward svc/openobserve 5080:5080 -n openobserveForwarding from 127.0.0.1:5080 -> 5080Forwarding from [::1]:5080 -> 5080
OpenObserve Login
使用上面指定的管理员邮箱和密码即可登录,然后就可以看到 OpenObserve 的主界面:
OpenObserve Web
因为现在还没有数据,所以页面中没有任何内容,在 ingestion 页面提供了 Logs、Metrics、Traces 数据的各种摄取方法:
这里我们可以先使用 JSON API 来加载一些示例日志数据来了解一下 OpenObserve 的使用方法。先使用下面命令下载示例日志数据:
$ curl -L-o k8slog_json.json.zip$ unzip k8slog_json.json.zip
然后使用下面命令将示例日志数据导入到 OpenObserve 中:
$ curl-i -u "[emailprotected]:root321"-d "@k8slog_json.json"HTTP/1.1 100 ContinueHTTP/1.1 200 OKcontent-length: 71vary: Origin, Access-Control-Request-Method, Access-Control-Request-Headerscontent-type: application/jsondate: Fri, 04 Aug 2023 10:46:46 GMT{"code":200,"status":[{"name":"default","successful":3846,"failed":0}]}%
收据导入成功后,刷新页面即可看到有数据了:
OpenObserve Web
在 Stream 页面可以看到我们导入的数据元信息:
Stream流
然后可以切换到Logs页面就可以看到日志数据了:
OpenObserve Logs
现在我们就可以去根据直接的需求去查询日志了,常用的一些查询语法如所示:
当然除了日志之外,OpenObserve 还支持指标和追踪数据,这里就不再演示了,有兴趣的可以自己去尝试一下。
这里我们只是简单的演示了一下 OpenObserve 的日志方面的使用方法,后续我们可以使用 Fluentbit、Vector 之类的工具来将 Kubernetes 集群中的日志数据发送到 OpenObserve 中,敬请期待!
童趣的译文是什么?
我回忆幼小的时候,能睁大眼睛对着太阳,眼力足以看清极细的东西。 看到细小的东西,一定要仔细观察它的花纹。 所以我时常有观察物体本身以外的乐趣。 夏天的蚊群飞鸣声像雷一样,我私下里把它们比做鹤群在空中飞舞。 心中想象的景观是鹤,那么呈现在眼前或是成千、或是成百飞舞着的蚊子便果真(觉得它们)是鹤了。 仰起头来观看这种景象,脖颈因此都僵硬了。 (有时)我又把蚊子留在白色的蚊帐里,用加慢慢地喷它们,使它们冲着烟雾飞叫,当做青云鹤图来看,果真就像鹤在云头上高亢地鸣叫,令人高兴得连声赞好。 我常在高洼不平的土墙边,杂草丛生的花台旁,蹲下自己的身子,使身子和花台一样高,定睛细看。 把繁茂的杂草看作树林,把昆虫蚂蚁看成野兽,把泥土瓦砾突起的地方看成山丘,低洼的地方看成沟谷,想象自己在里面游历的情景,真感到心情舒畅,自得其乐。 一天,看见两只虫子在草丛间相斗,我观看这一情景兴趣正浓厚的时候,突然一个很大的东西像推开大山,撞倒大树一般地闯过来,原来是一只癞蛤蟆。 (蛤蟆)舌头一伸,两只虫子就全被吞进肚里。 我那时年纪还小,正看得出神,不禁哎呀地(惊叫)一声,感到害怕;心神安定下来,捉住蛤蟆,打了它几十鞭,把它赶到别的院子里去了。
“正如爱迪生所说的”,英文怎么表达?
Just as Edison said,...
有没有介绍法国大餐的用餐礼仪的英语版本
French Dining EtiquetteThe French are known for formality and politeness and at the table of the French, dining etiquette is absolutely a must. So how exactly do you go about having a meal with your French friends without offending? Keep reading to learn what to expect when dining out or in. Given that the French are so in love with their food and its preparation, it only follows naturally that one must also have impeccable table manners to accompany the fine wine and dining. But be forewarned that what may look like beautiful table manners in America--is not in fact beautiful table manners in France. What to do with your hands For some reason, it is very rude to keep your hands in your lap while you are dining in France. It is equally rude to keep your elbows on the table and so the American diner must keep hands visible but elbows not. Also remember that the French almost never eat with their hands. This includes so called finger foods. The Bread Bread will always be served with your meal. If you are dining formally, bread will get its own plate. However, if you have no bread plate, the bread rests on the table cloth and not on your plate. There is also a manner in which you should eat bread. While most Americans may take a bite out of the bread--the French consider taking a bite out of a whole slice of bread to be rather boorish. Instead, tear off the bread piece by piece. If you are using bread to soak up sauce, use it on the end of a fork. Follow the Leader The number one rule in French dining etiquette is to follow the host. If you do this and observe carefully, you will almost never err. This is particularly important when were talking about beginning to eat, drink or anything else. In general, it is polite to wait until everyone is served and then the host will begin eating. Likewise, if there is an aperitif served, you should wait until the host raises her glass before you raise yours and take a drink. Eating Out Following the tips above will save you from embarrassment and making too many faux pas. While the rules of eating dont change drastically when youre eating out, there are a few special considerations to remember. Service Compris In America, the standard tip in a nice restaurant is between 15% and 20%. In France, tips are included. This is a French law so that the tips can be assessed for taxation purposes. You will see a service compris on your bill--which will be 15% of your food. However, one nice thing is that the price of food includes both the tip and the tax. So for example, if you order a piece of cake for $5 and a cup of coffee for $3, your bill is $8--service compris! An Extra Tip? Who doesnt appreciate an extra tip? While you wont be considered rude for not tipping, it is an appreciative gesture to tip your waiter if he worked well. Something that many Americans dont realize is that the tip that is included in your bill does not necessarily go to the waiters. Whats more, the owner of the restaurant can, by law, keep some or all of the tip. So anything additional you leave the waiter goes directly to him. So how much should you tip if you decide to tip extra? There is no hard and fast rule about percentages but if you consider about 5% of your check, this is a nice way to show gratitude to your waiter if he was especially diligent. When to Eat It is worth mentioning that the French tend to eat on a very different time table. It is important to note because in the afternoon when museums and shops are closed, cafes are open. But if you dont eat lunch while you can--you will go hungry until about 8pm--French dinner time. Remember: Eat your breakfast early in the morning when all the boulangeries are open. Eat lunch some time between 12pm and 3pm. Dont plan on much site seeing between 12pm and 3pm. This is standard lunch hour and things will be closed. Plan on a late dinner. If you get too hungry before then, plan early to take a snack with you. Dining and Eating in France The best way to get along in France is to dine and eat the way the French do! Take later lunches and dinners, remember to follow the lead of your hostess and most of all observe the rules of politeness.
发表评论