Delays in pilot/JobAgent cycles #6106
Answered
by
marianne013
marianne013
asked this question in
Q&A
-
|
Technically this is @sfayer's question.... but there are no more waiting jobs, so I would expect it to exit after 10 cycles. It's now more than 2 h later. Doing a horrible strace hack we find: which looks a bit like a red herring. Simon reckons something is stuck, but what ? |
Beta Was this translation helpful? Give feedback.
Answered by
marianne013
May 25, 2022
Replies: 9 comments 7 replies
-
|
Can I see the meaningful part of the result of Also, when you say that "it works in v7r3", do you mean py2 or py3? |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Our v7r3 is all py3, at least we have the Python3Pilots = True flag set, that should do it, no ? |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
I don't know what that flag is doing, but the wrapper (PilotWrapper.py) has got these lines right at the top:
So from the code above, python 3 is not even preferred, if it exists (python might point to python 2, as it does, see below)
The flag above manifests itself in the set of flags constructed by the pilot (from the command line and a pilot.json file), and in my case it is:
haven't investigated when it kicks in, but definitely not at the beginning. |
Beta Was this translation helpful? Give feedback.
-
|
The reason behind the lines above is that the locally preferred python version is the one to be used. On some WNs there's no python command but only python3 or python2, and in this case python3 is preferred. |
Beta Was this translation helpful? Give feedback.
-
|
In a possibly related issue, Simon thinks that you are buffering the whole pilot output in here: |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
This is a good point, and something that @martynia was actually asked to look into at the last BiLD. |
Beta Was this translation helpful? Give feedback.
-
|
I'm concentrating on a remote logger. It is buffered, and the buffer size could be anything >= 1 line. It will be configurable, so once ready it should solve the issue with truncated logs. |
Beta Was this translation helpful? Give feedback.
-
|
@martynia You are running v7r3 which should run python3 pilots automatically unless you specifically tell them otherwise. (Which you don't.) |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
From the listing below, we see that it is running the Agent with python3, otherwise Python (==Python 2) |
Beta Was this translation helpful? Give feedback.
-
|
To answer Federico's question from yesterday. When the node sits there not running any jobs, but also not shutting down I see: |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
And here is the pilot log - note that there hasn't been an update for over 60 min: |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
This actually opens another issue. In my current remote logging system:
This does not addreess an issue when you want to flush a buffer after some period of time , not after n lines. This would help to debug even w/o setting n=1. JM |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
Would you manage to provide a fix for flushing the buffer every n seconds? (e.g. 600 seconds) |
Beta Was this translation helpful? Give feedback.
-
|
It is on my to-do list, but I would like to do it for the remote logger first. The classic logger serves me as an example, so I can see which lines should appear and in which order. It's easy to lose log lines when designing a self-flushing buffer. The remote logger is quite advanced now. |
Beta Was this translation helpful? Give feedback.
-
|
This is probably more one for @sfayer : (it's right now 16:09 UTC.) Any creative ideas anyone ? |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
@fstagni I've moved the delay in the pilot finishing to a separate issue: #6116 |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
marianne013
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@fstagni I've moved the delay in the pilot finishing to a separate issue: #6116
@martynia: Can you make a separate discussion for the logging stuff if needed ?