Replies: 5 comments 3 replies
-
(quickly, no time now). Why do you think they are not reliable? MemoryUsed parameter is generated with
While what you see in the pilot is calculated with
|
Beta Was this translation helpful? Give feedback.
-
Taking one random LHCb job: In Jobs parameters:
In pilot log:
I do not see discrepancies. BTW: result["Memory(kB)"] = int(psutil.virtual_memory()[1] / 1024) which is the available virtual memory. Are you running single or multi-core applications? |
Beta Was this translation helpful? Give feedback.
-
We are using single core applications, but we use :
I don't know if it makes a difference in getting memory informations. Anyway, I also don't see any discrepancies between Job Parameters and what is reported in the pilot log. However the MemoryUsed parameter does not correspond at all (by orders of magnitudes) to what we have measured for our application independentely. While the RSS computed by:
i.e. using psutil corresponds well to our measurements. As far as I understand the memory computed with psutil corresponds to a given process (the one associated to the job I guess), while:
gives the memory used by the caller, which in this case I guess is the JobWrapper. Is that correct?
which is fine. So I'm wondering if:
correctly takes into account the memory used by the process associated to the job or it takes something else. To give you an example for a given job we have RSS of the order of: RSS: 2887944.0 kb and MemoryUsed (kb): 69912.0 Can you please explain the large difference between these 2 values? |
Beta Was this translation helpful? Give feedback.
-
Few things:
|
Beta Was this translation helpful? Give feedback.
-
Thanks a lot for the PR. It looks fine to me. I just have a minor comment that I will put directly on the PR. Then, about using Pool as Local CE, if NumeberOfProcessors is set = 1, then there will be no jobs in parallel right? So on in this case the log shouldn't be messy. |
Beta Was this translation helpful? Give feedback.
-
Dear all,
As far as I know the 2 memory related job parameters: 'Memory', 'MemoryUsed'
are not reliable or at least I don't understand their meaning.
Can you please confirm?
What it seems a reliable information instead is the RSS information logged in the pilot output.
Can you also confirm?
Thank you.
Beta Was this translation helpful? Give feedback.
All reactions