Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recover from disconnects #17

Open
pushcx opened this issue Sep 18, 2024 · 3 comments
Open

Recover from disconnects #17

pushcx opened this issue Sep 18, 2024 · 3 comments
Labels
bug Something isn't working good first issue Good for newcomers

Comments

@pushcx
Copy link
Member

pushcx commented Sep 18, 2024

A couple times in the last few weeks mockturtle has lost its connection to Libera. It continues running but can’t reconnect.

Maybe the straightforward fix here is to detect the net connection has dropped and exit the process? I’d rather not build complicated reconnect logic, let a new process rebuild. It’s not so bad if we lose a single post along the way, though depending when and where the first exception happens, maybe the cache file won’t be updated so the new process will handle it?

Sep 16 19:25:28 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"defaultMeta":{"account":false},"message":"Activating scheduled tasks.","level":"info","timestamp":"2024-09-16 19:25:28"}
Sep 16 19:29:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"defaultMeta":{"account":false},"message":"Activating scheduled tasks.","level":"info","timestamp":"2024-09-16 19:29:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Client Ping timeout (120 seconds)","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Connection.end() connected=true with data=true had_error=true","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Connection.end() connected=true with data=false had_error=true","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport close() destroying","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport socketClose()","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Socket closed. was_connected=true safely_registered=true requested_disconnect=false","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Scheduling reconnect. Attempt: 1/3 Wait: 6894ms","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:53 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Connection.setEncoding() encoding=utf8","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:53"}
Sep 16 19:31:53 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport connect()","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:53"}
Sep 16 19:31:53 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport disposeSocket() connected=false","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:53"}
Sep 16 19:31:53 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport Connection.setEncoding() encoding=utf8","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:53"}
Sep 16 19:31:53 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport Connecting socket..","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:53"}
Sep 16 19:31:53 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport socketRawConnected()","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:53"}
Sep 16 19:31:54 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport socketFullyConnected()","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:54"}
Sep 16 19:31:54 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Socket fully connected","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:54"}
Sep 16 19:32:27 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport socketClose()","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:32:27"}
Sep 16 19:32:27 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Socket closed. was_connected=true safely_registered=false requested_disconnect=false","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:32:27"}
Sep 16 21:21:37 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"defaultMeta":{"account":"mockturtle"},"itemDate":"2024-09-16T21:21:01.000Z","itemGuid":"https://lobste.rs/s/mun2dn","lastSeen":"2024-09-16T21:21:01.000Z","level":"info","message":"Broadcasting story","timestamp":"2024-09-16 21:21:37"}
Sep 16 21:21:37 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"write() called when not connected","level":"debug","message":"Debug Event","timestamp":"2024-09-16 21:21:37"}
Sep 16 21:21:37 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"write() called when not connected","level":"debug","message":"Debug Event","timestamp":"2024-09-16 21:21:37"}

@pushcx pushcx added bug Something isn't working good first issue Good for newcomers labels Sep 18, 2024
@OscarFKE
Copy link
Contributor

OscarFKE commented Sep 18, 2024

I'm having a look through the code now, my first guess is that the bot is not responding to PING requests from the libera irc server. Have you updated the dependencies recently?

I have some free time tomorrow, I'll try my hand at updating the bot to gracefully exit the process on connection end.

@pushcx
Copy link
Member Author

pushcx commented Sep 18, 2024

It's been quite some time since we bumped dependencies on this bot. I don't know if Libera changed something about pings recently.

@pushcx
Copy link
Member Author

pushcx commented Sep 18, 2024

I asked in #libera-communities and an op said they don't believe anything has changed in this area.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

2 participants