Hi folks,
I've uploaded the ISL script and the bundle which is uses to our RAMADDA server:
http://ramadda.atmos.albany.edu:8080/repository/entry/show/Top/IDV+Bundles/5.0?entryid=8e5ddea5-4542-4e39-95e3-3fc1d8f20ca1
The iDV bundle was made with the 5.0 test release, so be sure you are using 5.0.
A few observations/questions:
1. I can confirm Julien's remarks regarding performance varying depending on
what the host system that runs the script is doing. Last night on my 16GB
MacBook Pro (with approx. 10 GB allocated via -Xmx to IDV), performance slowed
to a crawl after about 33 of the 61 frames had loaded ... "top" showed that the
memory usage had indeed reached the maximum. Tonight, RAM usage climbed to
about 5.5GB but remained steady after that, and all 61 frames loaded in a
timely fashion. This could be a function of both the host system and also the
character of the HRRR radar fields that were loaded tonight as opposed to last
night.
2. ISL question: If you look at the script, I have a foreach loop that
explicitly has each frame increment. Is there any provision in ISL that allows
one to set an integer variable and simply do a "while" loop with an increment
at the end of the loop? Or anything else that would avoid having to hard code
each frame number?
3. ISL question 2: One easy way to avoid the memory usage creeping up would be
to call the ISL script from a shell script, where I would just pass in the
frame # and do the looping of frame numbers in the shell script. However, I
see no way to pass a variable into an ISL script.
4. Since the .xidv file points to the latest available HRRR run, you may end up
with less than 61 frames. Not a problem, just beware if you see that only 5
frames load!
Thanks,
Kevin
_____________________________________________
Kevin Tyle, Systems Administrator
Dept. of Atmospheric & Environmental Sciences
University at Albany
Earth Science 235, 1400 Washington Avenue
Albany, NY 12222
Email: ktyle@xxxxxxxxxx
Phone: 518-442-4578
_____________________________________________
________________________________________
From: Julien Chastang <chastang@xxxxxxxx>
Sent: Monday, January 6, 2014 5:40 PM
To: <idv-steering@xxxxxxxxxxxxxxxx>; idvdevelopers@xxxxxxxxxxxxxxxx
Subject: Re: [idvdevelopers] 20131220: New test release
Don,
On 1/6/14 1:55 PM, Don Murray (NOAA Affiliate) wrote:
> Julien-
>
> On 1/6/14 1:07 PM, Julien Chastang wrote:
>> Ideally, what we really want is something that predicts if we are going
>> to reach IDV memory limitations ahead of time.
>
> In the past, the IDV didn't predict, but it did monitor and took steps
> to alleviate when memory was reaching close to the limit. It also took
> action to clean up memory proactively, like when all displays and data
> were removed. A lot of thought was put into that. It wasn't perfect,
> but seemed to work reasonably well. Apparently, the new paradigm of
> letting the JVM sort this out for itself isn't working in Kevin's case.
Garbage collection algorithms have evolved significantly (or have been
replaced entirely) since the code in question was originally written.
Again, obtaining the bundle will allow us to obtain accurate metrics on
IDV memory consumption.
Yuan also suggested being aware of what other processes/applications are
running on your machine. You may have better results if you actually
*lower* the amount of memory used via the Edit, Preference, System Tab.
The JVM will use all the memory you throw at it (and these days we are
throwing a lot, 80% of available), but if you have other apps running,
you could force page in/outs which could cause performance degradation
and even kernel panics. In fact, from looking at several bundles we have
anecdotal evidence suggesting you should never need more than ~3GB
allocated to the IDV. Making changes in the memory allocation heuristic
to not go beyond a certain hard limit (e.g., 3GB - 4GB) is something we
are studying.
Lastly, all of this will evolve as the JVM itself evolves. There will be
major improvement in Hotspot Java 8 including no more permgen in
addition to other important changes.
-Julien
>> Actually, this issue segues into a long-standing IDV steering committee
>> discussion. If possible, please pass along resource instense bundles to
>> the IDV team. We can put them through an analysis tool called a profiler
>> to gain a more accurate idea of the source of the problem.
>
> Hopefully, Kevin can share his isl script with you to test.
>
>> Yuan's forthcoming progressive resolution feature could also help here
>> as it is smarter about using limited computational resources.
>
> I doubt that's going to help Kevin who has probably already set up his
> bundle to subset over the area.
>
> Don
>
>> On 1/6/14 10:42 AM, Tyle, Kevin R wrote:
>>> Yes, at least in the ISL script, while monitoring memory use via top,
>>> the memory usage matches what the IDV shows as current memory usage.
>>> Once this number meets the -Xmx limit, my ISL script grinds to a halt
>>> (i.e., goes from ~ 4 sec per frame to over a minute, and then
>>> eventually freezes).
>>>
>>> _____________________________________________
>>> Kevin Tyle, Systems Administrator
>>> Dept. of Atmospheric & Environmental Sciences
>>> University at Albany
>>> Earth Science 235, 1400 Washington Avenue
>>> Albany, NY 12222
>>> Email: ktyle@xxxxxxxxxx
>>> Phone: 518-442-4578
>>> _____________________________________________
>>>
>>>
>>> -----Original Message-----
>>> From: Julien Chastang [mailto:chastang@xxxxxxxx]
>>> Sent: Monday, January 06, 2014 12:40 PM
>>> To: Tyle, Kevin R; Don.Murray@xxxxxxxx; Yuan Ho
>>> Cc: Bob Carp; Joleen Feltz; <idv-steering@xxxxxxxxxxxxxxxx>;
>>> idvdevelopers@xxxxxxxxxxxxxxxx
>>> Subject: Re: [idvdevelopers] 20131220: New test release
>>>
>>> Happy New Year All.
>>>
>>> Kevin,
>>>
>>> Focusing on those diagnostic numbers can be somewhat misleading
>>> (because the internal JVM memory management algorithms are changing
>>> and improving all the time). For example, just because you see a high
>>> current memory number does not mean there is a problem. It could just
>>> be that the JVM has decided not to employ one of several memory
>>> reclamation (garbage
>>> collection) strategies for whatever reason.
>>>
>>> The real question is are you seeing your IDV crashing? Is there a
>>> problem?
>>>
>>> Best,
>>>
>>> -Julien
>>>
>>>
>>> On 1/6/14 10:26 AM, Tyle, Kevin R wrote:
>>>> It's a definite issue, and it becomes a real problem when running an
>>>> ISL script that loops over all frames. I was hitting the 6GB limit
>>>> by frame 33 or so on one of my higher-RAM systems when running an ISL
>>>> script that creates US forecast radar plots from the HRRR over all 61
>>>> frames.
>>>>
>>>> _____________________________________________
>>>> Kevin Tyle, Systems Administrator
>>>> Dept. of Atmospheric & Environmental Sciences University at Albany
>>>> Earth Science 235, 1400 Washington Avenue Albany, NY 12222
>>>> Email: ktyle@xxxxxxxxxx
>>>> Phone: 518-442-4578
>>>> _____________________________________________
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Don Murray (NOAA Affiliate) [mailto:don.murray@xxxxxxxx]
>>>> Sent: Monday, January 06, 2014 12:24 PM
>>>> To: Tyle, Kevin R; Yuan Ho
>>>> Cc: <idv-steering@xxxxxxxxxxxxxxxx>; Bob Carp; Joleen Feltz;
>>>> idvdevelopers@xxxxxxxxxxxxxxxx
>>>> Subject: Re: [idvdevelopers] 20131220: New test release
>>>>
>>>> Kevin-
>>>>
>>>> I believe that the IDV used to do a garbage collection (i.e. clean up
>>>> memory) after the Remove All Displays and Data action, but I know
>>>> Julien and Sean did some work to turn off all garbage collection and
>>>> let the system clean up by itself. Ideally, it should do that, but
>>>> maybe that's not working as it should.
>>>>
>>>> I agree that it was nice to have the "how much you've used" to get an
>>>> idea of when to clear the cache or whether you can actually load in
>>>> that next display.
>>>>
>>>> Don
>>>>
>>>> On 1/6/14 8:05 AM, Tyle, Kevin R wrote:
>>>>> I've noticed that this test release does not free memory when I
>>>>> select "Remove All Displays and Data". Some times, but not most
>>>>> times, memory will get freed when I proceed to load in a different
>>>>> data source. Usually I have to completely restart the IDV to get
>>>>> memory use down.
>>>>>
>>>>> I've also noticed that the three memory use values in the lower left
>>>>> of the screen (i.e., the ones that appear when you toggle off the
>>>>> realtime clock) have been replaced by only two values (current
>>>>> use/upper limit of use) ... i.e., the maximum use in the current
>>>>> session value no longer appears. That was a nice feature ...
>>>>>
>>>>> --Kevin
>>>>>
>>>>> _____________________________________________
>>>>> Kevin Tyle, Systems Administrator
>>>>> Dept. of Atmospheric & Environmental Sciences University at Albany
>>>>> Earth Science 235, 1400 Washington Avenue Albany, NY 12222
>>>>> Email: ktyle@xxxxxxxxxx
>>>>> Phone: 518-442-4578
>>>>> _____________________________________________
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: idvdevelopers-bounces@xxxxxxxxxxxxxxxx
>>>>> [mailto:idvdevelopers-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Don
>>>>> Murray
>>>>> Sent: Friday, December 20, 2013 6:24 PM
>>>>> To: Yuan Ho
>>>>> Cc: <idv-steering@xxxxxxxxxxxxxxxx>; Bob Carp; Joleen Feltz;
>>>>> idvdevelopers@xxxxxxxxxxxxxxxx
>>>>> Subject: Re: [idvdevelopers] 20131220: New test release
>>>>>
>>>>> One note about this. If you made any bundles from the previous
>>>>> progressive resolution test release, they will probably no longer
>>>>> work (that's why it's called a test release ;-)). We had to make
>>>>> some significant changes to simplify the underlying code and it was
>>>>> not feasible to keep the old test bundles working. You will need to
>>>>> recreate them, which would be a good test of whether things work
>>>>> from scratch and from the new bundles.
>>>>>
>>>>> And, as some of you have found, you cannot necessarily use any
>>>>> bundles created under the test release with the official current
>>>>> release. But please test any bundles made with the official release
>>>>> (4.1) under 5.0.
>>>>>
>>>>> Don
>>>>>
>>>>> On 12/20/13 3:52 PM, Yuan Ho wrote:
>>>>>> Hi All,
>>>>>> I have a new test release here for the progressive
>>>>>> resolution and new image chooser. Please download from the
>>>>>> following link:
>>>>>>
>>>>>> http://www.unidata.ucar.edu/downloads/idv/test/index.jsp
>>>>>>
>>>>>>
>>>>>> The changes in the latest:
>>>>>>
>>>>>> 0) the release version is 5.0 for this test release
>>>>>> 1) remove the progressive resolution checkbox from subset panel, it
>>>>>> is now set as the user preference and can be changed in view drop
>>>>>> down menu
>>>>>> 2) the use display area is added as a property of the datasource in
>>>>>> the data source properties panel
>>>>>> 3) the show mag factor is added in the sidelegend
>>>>>> 4) most of previous bugs should be fixed.
>>>>>>
>>>>>> The design should be very close to be the final. My plan is to merge
>>>>>> this to the nightly release after the new year, and run more test
>>>>>> and prepare for the AMS short course with this version of IDV.
>>>>>>
>>>>>>
>>>>>> Yuan
>>>>> --
>>>>> Don Murray
>>>>> NOAA/ESRL/PSD and CIRES
>>>>> 303-497-3596
>>>>> http://www.esrl.noaa.gov/psd/people/don.murray/
>>>>>
>>>>> _______________________________________________
>>>>> idvdevelopers mailing list
>>>>> idvdevelopers@xxxxxxxxxxxxxxxx
>>>>> For list information, to unsubscribe, visit:
>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>>
>>>>>
>>>> --
>>>> Don Murray
>>>> NOAA/ESRL/PSD and CU-CIRES
>>>> 303-497-3596
>>>> http://www.esrl.noaa.gov/psd/people/don.murray/
>>>>
>>>> _______________________________________________
>>>> idvdevelopers mailing list
>>>> idvdevelopers@xxxxxxxxxxxxxxxx
>>>> For list information, to unsubscribe, visit:
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>
>
