TRACING-4752: Add OpenTelemetry-Collector as optional sub-package #4281

copejon · 2024-12-06T14:46:46Z

Which issue(s) this PR addresses:

Closes #

openshift-ci · 2024-12-06T14:46:50Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2024-12-06T14:47:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: copejon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [copejon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

packaging/rpm/microshift.spec

packaging/observability/opentelemetry-collector.yaml

packaging/rpm/microshift.spec

packaging/observability/microshift-observability.service

ggiguash · 2024-12-09T07:37:27Z

packaging/observability/microshift-observability.service

@@ -0,0 +1,27 @@
+[Unit]
+Description=MicroShift Observability
+BindsTo=microshift.service


Do we want to run the collector even when MicroShift fails?

I'd think yes. If MicroShift fails to start, the metrics and log data should still be collectable by the metrics/logging backend remotely.

packaging/observability/microshift-observability.service

ggiguash · 2024-12-09T07:49:41Z

/retitle NO-ISSUE: OpenTelemetry certificates and service for MicroShift

openshift-ci-robot · 2024-12-09T07:49:47Z

@copejon: This pull request explicitly references no jira issue.

In response to this:

Which issue(s) this PR addresses:

Closes #

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

packaging/observability/opentelemetry-collector.yaml

openshift-ci-robot · 2024-12-12T20:47:24Z

@copejon: This pull request references TRACING-4752 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

Which issue(s) this PR addresses:

Closes #

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

copejon · 2024-12-12T20:47:28Z

/jira refresh

openshift-ci-robot · 2024-12-12T20:47:31Z

@copejon: This pull request references TRACING-4752 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

ggiguash · 2025-01-23T08:30:03Z

packaging/rpm/microshift.spec

+Requires: opentelemetry-collector
+
+%description observability
+Deploys the Red Hat build of Opentelemetry-collector as a systemd service on host. MicroShift provides client


We need to be consistent in the naming case. Either fix this, or the Summary section, please.

Suggested change

Deploys the Red Hat build of Opentelemetry-collector as a systemd service on host. MicroShift provides client

Deploys the Red Hat build of OpenTelemetry-Collector as a systemd service on host. MicroShift provides client

ggiguash · 2025-01-23T08:32:37Z

packaging/rpm/microshift.spec

+certificates to permit access to the kube-apiserver metrics endpoints. If a user defined opentelemetry-collector exists
+at /etc/microshift/opentelemetry-collector.yaml, this config is used. Otherwise, a default config is provided. Note that
+the default configuration requires the backend endpoint be set by the user.


Suggested change

certificates to permit access to the kube-apiserver metrics endpoints. If a user defined opentelemetry-collector exists

at /etc/microshift/opentelemetry-collector.yaml, this config is used. Otherwise, a default config is provided. Note that

the default configuration requires the backend endpoint be set by the user.

certificates to permit access to the kube-apiserver metrics endpoints. If a user-defined configuration file exists

at /etc/microshift/opentelemetry-collector.yaml, this configuration is used. Otherwise, a default configuration is provided.

Note that the default configuration requires the backend endpoint be set by the user.

For the backend endpoint, should we be specific on what we expect users to set?
I mean, should we say exporters.otlp section must be edited by users?

Added more specific instructions

ggiguash · 2025-01-23T08:39:33Z

packaging/observability/opentelemetry-collector.yaml

+# EXAMPLE OTLP (Prometheus) ENDPOINT CONFIG
+# The otlp exporter requires an endpoint listening for OTLP connections. To prevent spamming the log with Go
+# stack traces, the exporter is disabled. The endpoint is not known at installation, thus a tire-kicking of the
+# microshift-observability package would result in stack traces spam in logs.


Let's think what we can do so that the logs are not "spammed" when the default configuration is used. It sounds as if we should copy this file with .example suffix so that users would have to explicitly rename the file when they enable the collector service.

In any case, the "style" of this comment should be reworded.

I've tweaked the comment and made it a little more informational

ggiguash · 2025-01-23T08:48:46Z

packaging/observability/microshift-observability.service

@@ -0,0 +1,20 @@
+[Unit]
+Description=MicroShift Observability
+After=microshift.service


Should we use ConditionPathExists here for all the files the service expects to have before it starts?

The opentelemetry-collector performs that check for us each time it starts.

Right, but the point of the condition in systemd is not to attempt starting the service if the path does no exist.

Could this help to avoid unnecessary restarts?

ggiguash · 2025-01-23T08:53:35Z

packaging/observability/microshift-observability.service

+
+# It takes a bit for the certs to be created. This service will reach it's burst limit almost immediately, pretty much
+# guaranteeing that it will reach the restart limit before it can possibly succeed.
+RestartSec=200ms


Do we really need this? We've configured the service to start After microshift, so microshift must report readiness to systemd before the current service startup is attempted. MicroShift only reports readiness after creating all certificates.
What am I missing?

In earlier tests this was necessary to keep the service from crash looping, but that doesn't seem to be an issue in the latest opentelemetry-collector. Will remove

packaging/observability/microshift-observability.service

ggiguash · 2025-01-28T16:57:52Z

packaging/observability/opentelemetry-collector.yaml

+    auth_type: tls
+    ca_file: /etc/pki/microshift-opentelemetry-collector-client/client-ca.crt
+    key_file: /etc/pki/microshift-opentelemetry-collector-client/client.key
+    cert_file: /etc/pki/microshift-opentelemetry-collector-client/client.crt


These paths need to be updated too -> /var/lib/microshift/..../

ggiguash · 2025-01-28T17:01:59Z

packaging/rpm/microshift.spec

+certificates to permit access to the kube-apiserver metrics endpoints. If a user defined Opentelemetry-Collector exists
+at /etc/microshift/opentelemetry-collector.yaml, this config is used. Otherwise, a default config is provided. Note that
+the default configuration requires the backend endpoint be set by the user. The otlp export must also be specified as
+ .service.pipelines.$RECIEVER.exporter: "otlp".  The specification for the otlp config is:


Let's not use shortened words because it's a user-facing RPM description.

ggiguash · 2025-01-30T15:04:46Z

packaging/rpm/microshift.spec

+Requires: opentelemetry-collector
+
+%description observability
+Deploys the Red Hat build of Opentelemetry-Collector as a systemd service on host. MicroShift provides client


Please, fix the case of Opentelemety -> OpenTelemetry to make it consistent with the summary text.

…elemetry-collector, preconfigured for microshift

copejon · 2025-02-18T22:10:07Z

I've gotten the test suite into a functioning state locally. Up next: t-shirt sized configs to enable different levels of data collecting.

add test for error in log handle uninstalling observability systemd units gracefully Signed-off-by: Jon Cope <[email protected]>

openshift-ci · 2025-02-21T00:00:57Z

@copejon: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-tests-bootc	`5845981`	link	true	`/test e2e-aws-tests-bootc`
ci/prow/e2e-aws-tests	`5845981`	link	true	`/test e2e-aws-tests`
ci/prow/e2e-aws-tests-bootc-arm	`5845981`	link	true	`/test e2e-aws-tests-bootc-arm`
ci/prow/verify	`5845981`	link	true	`/test verify`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 6, 2024

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 6, 2024

ggiguash reviewed Dec 9, 2024

View reviewed changes

openshift-ci bot changed the title ~~No issue generate otel cert~~ NO-ISSUE: OpenTelemetry certificates and service for MicroShift Dec 9, 2024

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Dec 9, 2024

pavolloffay reviewed Dec 12, 2024

View reviewed changes

packaging/observability/opentelemetry-collector.yaml Show resolved Hide resolved

copejon changed the title ~~NO-ISSUE: OpenTelemetry certificates and service for MicroShift~~ TRACING-4752: Add OpenTelemetry-Collector as optional sub-package Dec 12, 2024

copejon force-pushed the no-issue-generate-otel-cert branch from fa4f579 to fede276 Compare December 12, 2024 21:05

copejon marked this pull request as ready for review January 21, 2025 16:45

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 21, 2025

openshift-ci bot requested review from agullon and eslutsky January 21, 2025 16:46

copejon force-pushed the no-issue-generate-otel-cert branch from 2042714 to ccfea22 Compare January 22, 2025 23:20

ggiguash reviewed Jan 23, 2025

View reviewed changes

packaging/observability/microshift-observability.service Outdated Show resolved Hide resolved

ggiguash reviewed Jan 28, 2025

View reviewed changes

copejon force-pushed the no-issue-generate-otel-cert branch from aee833a to ad4892d Compare January 30, 2025 13:13

ggiguash reviewed Jan 30, 2025

View reviewed changes

copejon force-pushed the no-issue-generate-otel-cert branch from 33de178 to 2996e90 Compare February 11, 2025 19:58

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 13, 2025

copejon force-pushed the no-issue-generate-otel-cert branch from 6c3feda to 024edb1 Compare February 13, 2025 22:21

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 13, 2025

copejon force-pushed the no-issue-generate-otel-cert branch 3 times, most recently from 6562511 to 0241f48 Compare February 17, 2025 19:40

add microshift sub-package to enable the optional deployment of opent…

e7136a4

…elemetry-collector, preconfigured for microshift

copejon force-pushed the no-issue-generate-otel-cert branch from cc06c31 to e7136a4 Compare February 18, 2025 22:07

add k8s event pipeline

5845981

add test for error in log handle uninstalling observability systemd units gracefully Signed-off-by: Jon Cope <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TRACING-4752: Add OpenTelemetry-Collector as optional sub-package #4281

TRACING-4752: Add OpenTelemetry-Collector as optional sub-package #4281

copejon commented Dec 6, 2024

openshift-ci bot commented Dec 6, 2024

openshift-ci bot commented Dec 6, 2024

ggiguash Dec 9, 2024

copejon Jan 22, 2025

ggiguash commented Dec 9, 2024

openshift-ci-robot commented Dec 9, 2024

openshift-ci-robot commented Dec 12, 2024 •

edited by openshift-ci bot

Loading

copejon commented Dec 12, 2024

openshift-ci-robot commented Dec 12, 2024 •

edited by openshift-ci bot

Loading

ggiguash Jan 23, 2025

ggiguash Jan 23, 2025

ggiguash Jan 23, 2025

copejon Jan 23, 2025

ggiguash Jan 23, 2025

ggiguash Jan 23, 2025 •

edited

Loading

copejon Jan 23, 2025

ggiguash Jan 23, 2025

copejon Jan 23, 2025

ggiguash Jan 27, 2025

ggiguash Jan 27, 2025

ggiguash Jan 23, 2025

copejon Jan 23, 2025

ggiguash Jan 28, 2025

ggiguash Jan 28, 2025

ggiguash Jan 30, 2025

copejon commented Feb 18, 2025

openshift-ci bot commented Feb 21, 2025

	Deploys the Red Hat build of Opentelemetry-collector as a systemd service on host. MicroShift provides client
	Deploys the Red Hat build of OpenTelemetry-Collector as a systemd service on host. MicroShift provides client

TRACING-4752: Add OpenTelemetry-Collector as optional sub-package #4281

Are you sure you want to change the base?

TRACING-4752: Add OpenTelemetry-Collector as optional sub-package #4281

Conversation

copejon commented Dec 6, 2024

openshift-ci bot commented Dec 6, 2024

openshift-ci bot commented Dec 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggiguash commented Dec 9, 2024

openshift-ci-robot commented Dec 9, 2024

openshift-ci-robot commented Dec 12, 2024 • edited by openshift-ci bot Loading

copejon commented Dec 12, 2024

openshift-ci-robot commented Dec 12, 2024 • edited by openshift-ci bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggiguash Jan 23, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

copejon commented Feb 18, 2025

openshift-ci bot commented Feb 21, 2025

openshift-ci-robot commented Dec 12, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Dec 12, 2024 •

edited by openshift-ci bot

Loading

ggiguash Jan 23, 2025 •

edited

Loading