Kevin,
A few comments:
Ideally, what we really want is something that predicts if we are going
to reach IDV memory limitations ahead of time.
Actually, this issue segues into a long-standing IDV steering committee
discussion. If possible, please pass along resource instense bundles to
the IDV team. We can put them through an analysis tool called a profiler
to gain a more accurate idea of the source of the problem.
Yuan's forthcoming progressive resolution feature could also help here
as it is smarter about using limited computational resources.
-Julien
On 1/6/14 10:42 AM, Tyle, Kevin R wrote:
Yes, at least in the ISL script, while monitoring memory use via top, the
memory usage matches what the IDV shows as current memory usage. Once this
number meets the -Xmx limit, my ISL script grinds to a halt (i.e., goes from ~
4 sec per frame to over a minute, and then eventually freezes).
_____________________________________________
Kevin Tyle, Systems Administrator
Dept. of Atmospheric & Environmental Sciences
University at Albany
Earth Science 235, 1400 Washington Avenue
Albany, NY 12222
Email: ktyle@xxxxxxxxxx
Phone: 518-442-4578
_____________________________________________
-----Original Message-----
From: Julien Chastang [mailto:chastang@xxxxxxxx]
Sent: Monday, January 06, 2014 12:40 PM
To: Tyle, Kevin R; Don.Murray@xxxxxxxx; Yuan Ho
Cc: Bob Carp; Joleen Feltz; <idv-steering@xxxxxxxxxxxxxxxx>;
idvdevelopers@xxxxxxxxxxxxxxxx
Subject: Re: [idvdevelopers] 20131220: New test release
Happy New Year All.
Kevin,
Focusing on those diagnostic numbers can be somewhat misleading (because the
internal JVM memory management algorithms are changing and improving all the
time). For example, just because you see a high current memory number does not
mean there is a problem. It could just be that the JVM has decided not to
employ one of several memory reclamation (garbage
collection) strategies for whatever reason.
The real question is are you seeing your IDV crashing? Is there a problem?
Best,
-Julien
On 1/6/14 10:26 AM, Tyle, Kevin R wrote:
It's a definite issue, and it becomes a real problem when running an ISL script
that loops over all frames. I was hitting the 6GB limit by frame 33 or so on
one of my higher-RAM systems when running an ISL script that creates US
forecast radar plots from the HRRR over all 61 frames.
_____________________________________________
Kevin Tyle, Systems Administrator
Dept. of Atmospheric & Environmental Sciences University at Albany
Earth Science 235, 1400 Washington Avenue Albany, NY 12222
Email: ktyle@xxxxxxxxxx
Phone: 518-442-4578
_____________________________________________
-----Original Message-----
From: Don Murray (NOAA Affiliate) [mailto:don.murray@xxxxxxxx]
Sent: Monday, January 06, 2014 12:24 PM
To: Tyle, Kevin R; Yuan Ho
Cc: <idv-steering@xxxxxxxxxxxxxxxx>; Bob Carp; Joleen Feltz;
idvdevelopers@xxxxxxxxxxxxxxxx
Subject: Re: [idvdevelopers] 20131220: New test release
Kevin-
I believe that the IDV used to do a garbage collection (i.e. clean up
memory) after the Remove All Displays and Data action, but I know Julien and
Sean did some work to turn off all garbage collection and let the system clean
up by itself. Ideally, it should do that, but maybe that's not working as it
should.
I agree that it was nice to have the "how much you've used" to get an idea of
when to clear the cache or whether you can actually load in that next display.
Don
On 1/6/14 8:05 AM, Tyle, Kevin R wrote:
I've noticed that this test release does not free memory when I select "Remove All
Displays and Data". Some times, but not most times, memory will get freed when I
proceed to load in a different data source. Usually I have to completely restart the IDV
to get memory use down.
I've also noticed that the three memory use values in the lower left of the
screen (i.e., the ones that appear when you toggle off the realtime clock) have
been replaced by only two values (current use/upper limit of use) ... i.e., the
maximum use in the current session value no longer appears. That was a nice
feature ...
--Kevin
_____________________________________________
Kevin Tyle, Systems Administrator
Dept. of Atmospheric & Environmental Sciences University at Albany
Earth Science 235, 1400 Washington Avenue Albany, NY 12222
Email: ktyle@xxxxxxxxxx
Phone: 518-442-4578
_____________________________________________
-----Original Message-----
From: idvdevelopers-bounces@xxxxxxxxxxxxxxxx
[mailto:idvdevelopers-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Don
Murray
Sent: Friday, December 20, 2013 6:24 PM
To: Yuan Ho
Cc: <idv-steering@xxxxxxxxxxxxxxxx>; Bob Carp; Joleen Feltz;
idvdevelopers@xxxxxxxxxxxxxxxx
Subject: Re: [idvdevelopers] 20131220: New test release
One note about this. If you made any bundles from the previous progressive
resolution test release, they will probably no longer work (that's why it's
called a test release ;-)). We had to make some significant changes to
simplify the underlying code and it was not feasible to keep the old test
bundles working. You will need to recreate them, which would be a good test of
whether things work from scratch and from the new bundles.
And, as some of you have found, you cannot necessarily use any bundles created
under the test release with the official current release. But please test any
bundles made with the official release (4.1) under 5.0.
Don
On 12/20/13 3:52 PM, Yuan Ho wrote:
Hi All,
I have a new test release here for the progressive
resolution and new image chooser. Please download from the following link:
http://www.unidata.ucar.edu/downloads/idv/test/index.jsp
The changes in the latest:
0) the release version is 5.0 for this test release
1) remove the progressive resolution checkbox from subset panel, it
is now set as the user preference and can be changed in view drop
down menu
2) the use display area is added as a property of the datasource in
the data source properties panel
3) the show mag factor is added in the sidelegend
4) most of previous bugs should be fixed.
The design should be very close to be the final. My plan is to merge
this to the nightly release after the new year, and run more test
and prepare for the AMS short course with this version of IDV.
Yuan
--
Don Murray
NOAA/ESRL/PSD and CIRES
303-497-3596
http://www.esrl.noaa.gov/psd/people/don.murray/
_______________________________________________
idvdevelopers mailing list
idvdevelopers@xxxxxxxxxxxxxxxx
For list information, to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
Don Murray
NOAA/ESRL/PSD and CU-CIRES
303-497-3596
http://www.esrl.noaa.gov/psd/people/don.murray/
_______________________________________________
idvdevelopers mailing list
idvdevelopers@xxxxxxxxxxxxxxxx
For list information, to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
