SUPERVISOR - Failed to get logs using advanced API (NEW - NOT THE SAME) #5606

RafAustralia · 2025-02-05T22:50:32Z

Describe the issue you are experiencing

Hi all

I have read through both previous issues and cases which simply refer to each other and are now closed and limited...

This issue has not been solved from research I have done.

I was told this issue is a similar issue to what happens when a docker is installed, but now I have this issue with bare metal HAOS install.

What type of installation are you running?

Home Assistant OS

Which operating system are you running on?

Home Assistant Operating System

Steps to reproduce the issue

At random times supervisor logs stop and supervisor crashes all together
2.We have rebuilt new NVME 2 Drive and new HAOS and reinstated the backup - same or very similar issue occurs.
When supervisor crashes - system cannot be accessed, nor rebooted nor diagnosed in any other way but via ssh
Happens now around once weekly...
Might be related to Websockets, might be related to supervisors connectivity all together.
It appears there is an issue with obtaining advanced logs when call is being issued to get these... and timouts occur perhaps...
...

Anything in the Supervisor logs that might be useful for us?

copy of the error, since rebuild last night:

2025-02-05 21:52:41.298 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state running
2025-02-05 21:52:41.298 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
2025-02-05 22:06:30.107 ERROR (MainThread) [supervisor.api] Failed to get supervisor logs using advanced_logs API
Traceback (most recent call last):
  File "/usr/src/supervisor/supervisor/api/host.py", line 262, in advanced_logs_handler
    await response.prepare(request)
  File "/usr/local/lib/python3.13/site-packages/aiohttp/web_response.py", line 453, in prepare
    return await self._start(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/aiohttp/web_response.py", line 461, in _start
    await self._write_headers()
  File "/usr/local/lib/python3.13/site-packages/aiohttp/web_response.py", line 538, in _write_headers
    await writer.write_headers(status_line, self._headers)
  File "/usr/local/lib/python3.13/site-packages/aiohttp/http_writer.py", line 152, in write_headers
    self._write(buf)
    ~~~~~~~~~~~^^^^^
  File "/usr/local/lib/python3.13/site-packages/aiohttp/http_writer.py", line 81, in _write
    raise ClientConnectionResetError("Cannot write to closing transport")
aiohttp.client_exceptions.ClientConnectionResetError: Cannot write to closing transport
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/src/supervisor/supervisor/api/__init__.py", line 403, in get_supervisor_logs
    return await self._api_host.advanced_logs_handler(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        *args, identifier="hassio_supervisor", **kwargs
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/src/supervisor/supervisor/api/host.py", line 269, in advanced_logs_handler
    raise APIError(
        "Connection reset when trying to fetch data from systemd-journald."
    ) from ex
supervisor.exceptions.APIError: Connection reset when trying to fetch data from systemd-journald.
2025-02-05 22:09:12.617 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
2025-02-05 22:39:17.531 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
2025-02-05 22:52:41.301 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state running
2025-02-05 22:52:41.301 INFO (MainThread) [supervisor.resolution.checks.base] Run check for

System Health information

Home Assistant Supervisor
Host operating system | Home Assistant OS 14.2

Supervisor diagnostics

No response

Additional information

I would love if this was not just referred to other two issues and closed off... as I cannot find any related answer to the problem.
System is bran new and fresh - only 12 hours old now, since rebuild...
Network doesnt seem to be an issue, we have gone through it yesterday,
I run unify, and over 60 tuya devices communicating on that network and not timing out...

We have removed vscode as it was misbehaving badly and landing a lot of errors...
supervisor appears to be the main and only remaining source of crashes now...
I am not quite sure what drives these though

Any help would be amazing - please.

Thank you in advance - much appreciated.

@purcell-lab
@agners

Raf

agners · 2025-02-06T10:10:40Z

Can you share the host logs of that system as well?

When supervisor crashes - system cannot be accessed, nor rebooted nor diagnosed in any other way but via ssh

I wonder if Supervisor crash is just a symptom of a bigger issue. To me it sounds as if the system hangs completely on I/O or something.

What do you mean by system cannot be accessed? The Home Assistant Core frontend is also not available?

RafAustralia · 2025-02-06T11:07:47Z

Hi Stefan.

Thank u so much for saying hi.

Its bizzare type of behaviour.
Initially we thought maybe u it was too hot (once) then we saw few vscode errors and we deleted it
It did not help. Or nor enough.

Still crashed.
When it crashes… no directories are available. No backups. Not reboot. No logs. No advanced logs. Just ssh only. So to me system fully crashed.
Supervisor could be restarted via ssh but it does not enable reboot from gui only safe mode if lucky.
So best way to reboot is ssh host reboot. Then it comes back.

Ive left it alone today. And I can say logs are all green. Nothing happened since 10pm last night.
But I had no chance to do anything. Just a busy day. Unit running on full auto.

When things happen often message relates to aiohttp or websocket.api or journald which all in all looks like a timeouts or rejections in connections.

Again its been ok today but we are wondering what could be causing these.

These crashes are extreme. Sudden. Unexpected.

Finally. Brand new ssd. Brand new haos. And restored backup. One crash pretty much straight after restore….

Ill try to give u any logs u need if u r interested trying to conquer this issue. Just ket me know which parts specifically would help u. I do have screenshots if crashes from the past if these help

I am once more very grateful.

Raf

Failure404 · 2025-02-06T14:37:40Z

I am getting the very same error on a supervised debian 12 installation with Hass 2025.2.0 and Supervisor 2025.2.0.

Additionally I get another random occuring error since Supervisor 2025.2.0. I can't tell what triggers the error since it happens in intervals of 5 minutes to several hours and I don't think it is related to logs, but might be related to a bigger problem.

Today, even the whole system rebooted itself after it apparently crashed - I've never seen that before. I can't find anything suspicious in any log, except the OP's error and the one below coming from Supervisor.

Exception ignored in: <http.client.HTTPResponse object at 0x7f5f07f31b70>
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/http/client.py", line 432, in close
    super().close() # set "closed" flag
  File "/usr/local/lib/python3.13/http/client.py", line 445, in flush
    self.fp.flush()
ValueError: I/O operation on closed file.

RafAustralia · 2025-02-10T21:22:37Z

Can you share the host logs of that system as well?

When supervisor crashes - system cannot be accessed, nor rebooted nor diagnosed in any other way but via ssh

I wonder if Supervisor crash is just a symptom of a bigger issue. To me it sounds as if the system hangs completely on I/O or something.

What do you mean by system cannot be accessed? The Home Assistant Core frontend is also not available?

.........................................................................................................................................................................

First crash since rebuild...
benign task... was creating a theme... half way through it - it decided to crash....

throughts?

application
2025-02-11 07:13:56.034 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/aiohttp/client_proto.py", line 92, in connection_lost
    uncompleted = self._parser.feed_eof()
  File "aiohttp/_http_parser.pyx", line 508, in aiohttp._http_parser.HttpParser.feed_eof
aiohttp.http_exceptions.TransferEncodingError: 400, message:
  Not enough data for satisfy transfer length header.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/aiohttp/web_protocol.py", line 480, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/aiohttp/web_app.py", line 569, in _handle
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 199, in block_bad_requests
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 215, in system_validation
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 285, in token_validation
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 298, in core_proxy
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/utils.py", line 108, in wrap_api
    msg_data = await method(api, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/host.py", line 279, in advanced_logs
    return await self.advanced_logs_handler(request, identifier, follow)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/host.py", line 257, in advanced_logs_handler
    async for cursor, line in journal_logs_reader(resp, log_formatter):
    ...<9 lines>...
            await response.write(line.encode("utf-8") + b"\n")
  File "/usr/src/supervisor/supervisor/utils/systemd_journal.py", line 80, in journal_logs_reader
    line = await resp.content.readuntil(b"\n")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/aiohttp/streams.py", line 386, in readuntil
    await self._wait("readuntil")
  File "/usr/local/lib/python3.13/site-packages/aiohttp/streams.py", line 347, in _wait
    await waiter
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed: <TransferEncodingError: 400, message='Not enough data for satisfy transfer length header.'>
2025-02-11 07:13:56.036 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/aiohttp/client_proto.py", line 92, in connection_lost
    uncompleted = self._parser.feed_eof()
  File "aiohttp/_http_parser.pyx", line 508, in aiohttp._http_parser.HttpParser.feed_eof
aiohttp.http_exceptions.TransferEncodingError: 400, message:
  Not enough data for satisfy transfer length header.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/aiohttp/web_protocol.py", line 480, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/aiohttp/web_app.py", line 569, in _handle
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 199, in block_bad_requests
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 215, in system_validation
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 285, in token_validation
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 298, in core_proxy
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/utils.py", line 108, in wrap_api
    msg_data = await method(api, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/host.py", line 279, in advanced_logs
    return await self.advanced_logs_handler(request, identifier, follow)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/host.py", line 257, in advanced_logs_handler
    async for cursor, line in journal_logs_reader(resp, log_formatter):
    ...<9 lines>...
            await response.write(line.encode("utf-8") + b"\n")
  File "/usr/src/supervisor/supervisor/utils/systemd_journal.py", line 80, in journal_logs_reader
    line = await resp.content.readuntil(b"\n")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/aiohttp/streams.py", line 386, in readuntil
    await self._wait("readuntil")
  File "/usr/local/lib/python3.13/site-packages/aiohttp/streams.py", line 347, in _wait
    await waiter
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed: <TransferEncodingError: 400, message='Not enough data for satisfy transfer length header.'>
2025-02-11 07:13:56.038 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/aiohttp/client_proto.py", line 92, in connection_lost
    uncompleted = self._parser.feed_eof()
  File "aiohttp/_http_parser.pyx", line 508, in aiohttp._http_parser.HttpParser.feed_eof
aiohttp.http_exceptions.TransferEncodingError: 400, message:
  Not enough data for satisfy transfer length header.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/aiohttp/web_protocol.py", line 480, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/aiohttp/web_app.py", line 569, in _handle
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 199, in block_bad_requests
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 215, in system_validation
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 285, in token_validation
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/middleware/security.py", line 298, in core_proxy
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/utils.py", line 108, in wrap_api
    msg_data = await method(api, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/host.py", line 279, in advanced_logs
    return await self.advanced_logs_handler(request, identifier, follow)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/api/host.py", line 257, in advanced_logs_handler
    async for cursor, line in journal_logs_reader(resp, log_formatter):
    ...<9 lines>...
            await response.write(line.encode("utf-8") + b"\n")
  File "/usr/src/supervisor/supervisor/utils/systemd_journal.py", line 80, in journal_logs_reader
    line = await resp.content.readuntil(b"\n")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/aiohttp/streams.py", line 386, in readuntil
    await self._wait("readuntil")
  File "/usr/local/lib/python3.13/site-packages/aiohttp/streams.py", line 347, in _wait
    await waiter
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed: <TransferEncodingError: 400, message='Not enough data for satisfy transfer length header.'>
s6-rc: info: service legacy-services: stopping
2025-02-11 07:13:56.093 INFO (MainThread) [supervisor.misc.scheduler] Shutting down scheduled tasks
2025-02-11 07:13:56.093 INFO (MainThread) [supervisor.docker.monitor] Stopped docker events monitor
2025-02-11 07:13:56.093 INFO (MainThread) [supervisor.api] Stopping API on 172.30.32.2
[21:13:56] INFO: Watchdog restart after closing

RafAustralia · 2025-02-20T21:37:14Z

has anyone had any more success with all of this? this post has gone quiet....

RafAustralia added the bug label Feb 5, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SUPERVISOR - Failed to get logs using advanced API (NEW - NOT THE SAME) #5606

SUPERVISOR - Failed to get logs using advanced API (NEW - NOT THE SAME) #5606

RafAustralia commented Feb 5, 2025

agners commented Feb 6, 2025

RafAustralia commented Feb 6, 2025

Failure404 commented Feb 6, 2025 •

edited

Loading

RafAustralia commented Feb 10, 2025

RafAustralia commented Feb 20, 2025

SUPERVISOR - Failed to get logs using advanced API (NEW - NOT THE SAME) #5606

SUPERVISOR - Failed to get logs using advanced API (NEW - NOT THE SAME) #5606

Comments

RafAustralia commented Feb 5, 2025

Describe the issue you are experiencing

What type of installation are you running?

Which operating system are you running on?

Steps to reproduce the issue

Anything in the Supervisor logs that might be useful for us?

System Health information

Supervisor diagnostics

Additional information

agners commented Feb 6, 2025

RafAustralia commented Feb 6, 2025

Failure404 commented Feb 6, 2025 • edited Loading

RafAustralia commented Feb 10, 2025

RafAustralia commented Feb 20, 2025

Failure404 commented Feb 6, 2025 •

edited

Loading