fix(telemetry): raise stdout StreamHandlers to WARN when LoggingHandler is installed - #347
Closed
jeanscherf wants to merge 2 commits into
Closed
jeanscherf wants to merge 2 commits into
jeanscherf wants to merge 2 commits into
Conversation
…ords in production LoggingHandler was created with no level (NOTSET), so if an application set the root logger to DEBUG, every framework debug trace — httpx connection events, OTel SDK internals, etc. — flowed into the production OTLP pipeline. Set level=logging.INFO on the handler so the SDK always enforces a minimum severity regardless of the application's root logger configuration.
…er is installed When both a StreamHandler and an OTel LoggingHandler are active on the root logger at INFO, every log record hits stdout twice — once via the StreamHandler and once via any ConsoleLogExporter in the OTel pipeline. The filelog receiver on the OTel collector picks up both copies, doubling CLS queue pressure and contributing to collector memory spikes. After installing the LoggingHandler, raise any existing StreamHandlers on the root logger to WARN so INFO+ flows exclusively via OTLP with full trace correlation. Stdout still carries warnings and errors for kubectl logs visibility. FileHandler and other StreamHandler subclasses are not affected. Also keeps INFO floor on the LoggingHandler itself to prevent DEBUG records from reaching production OTLP pipelines regardless of root logger level.
Member
Author
|
Superseded by a cleaner approach — the INFO floor on the LoggingHandler was wrong as it blocks developer DEBUG logs from reaching CLS. Reopening as a focused fix that only raises stdout StreamHandlers to WARN to prevent duplication. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
StreamHandlerand an OTelLoggingHandlerare active on the root logger at INFO, every log record hits stdout twice: once via theStreamHandlerand once via anyConsoleLogExporterin the OTel pipelineLoggingHandler, raise any existingStreamHandlerson the root logger toWARNso INFO+ flows exclusively via OTLP with full trace correlationkubectl logsvisibilityFileHandlerand otherStreamHandlersubclasses are not affected (exact type check)INFOfloor on theLoggingHandleritself to prevent DEBUG records reaching production OTLP pipelines regardless of root logger levelTest plan
TestRaiseStreamHandlersToWarn::test_raises_stream_handler_below_warn— StreamHandler at DEBUG is raised to WARNTestRaiseStreamHandlersToWarn::test_does_not_lower_stream_handler_above_warn— StreamHandler already at ERROR is not loweredTestRaiseStreamHandlersToWarn::test_ignores_file_handler_subclass— FileHandler is not touchedTestMakeLoggingHandler::test_handler_level_is_info— LoggingHandler level is INFOuv run pytest tests/core/unit/telemetry/test_provider.py::TestMakeLoggingHandler tests/core/unit/telemetry/test_provider.py::TestRaiseStreamHandlersToWarn -v