Re: [idvdevelopers] 20131220: New test release

Kevin,

Stepping back a moment, Yuan and I looked at your ISL script and we think we have a better solution for you. Please give the attached ISL a try and see if it better suits your needs.

Also this is something we will implement soon in the IDV, but set your memory to ~3GB via the Edit, Preferences.

Please keep us up-to-date on your progress.

-Julien

On 1/6/14 6:34 PM, Tyle, Kevin R wrote:
Hi folks,

I've uploaded the ISL script and the bundle which is uses to our RAMADDA server:

http://ramadda.atmos.albany.edu:8080/repository/entry/show/Top/IDV+Bundles/5.0?entryid=8e5ddea5-4542-4e39-95e3-3fc1d8f20ca1

The iDV bundle was made with the 5.0 test release, so be sure you are using 5.0.

A few observations/questions:

1. I can confirm Julien's remarks regarding performance varying depending on what the 
host system that runs the script is doing.  Last night on my 16GB MacBook Pro (with 
approx. 10 GB allocated via -Xmx to IDV), performance slowed to a crawl after about 33 of 
the 61 frames had loaded ... "top" showed that the memory usage had indeed 
reached the maximum.  Tonight, RAM usage climbed to about 5.5GB but remained steady after 
that, and all 61 frames loaded in a timely fashion.  This could be a function of both the 
host system and also the character of the HRRR radar fields that were loaded tonight as 
opposed to last night.

2. ISL question:  If you look at the script, I have a foreach loop that explicitly has 
each frame increment.  Is there any provision in ISL that allows one to set an integer 
variable and simply do a "while" loop with an increment at the end of the loop? 
 Or anything else that would avoid having to hard code each frame number?

3. ISL question 2:  One easy way to avoid the memory usage creeping up would be 
to call the ISL script from a shell script, where I would just pass in the 
frame # and do the looping of frame numbers in the shell script.  However, I 
see no way to pass a variable into an ISL script.

4. Since the .xidv file points to the latest available HRRR run, you may end up 
with less than 61 frames.  Not a problem, just beware if you see that only 5 
frames load!

Thanks,

Kevin

_____________________________________________
Kevin Tyle, Systems Administrator
Dept. of Atmospheric & Environmental Sciences
University at Albany
Earth Science 235, 1400 Washington Avenue
Albany, NY 12222
Email: ktyle@xxxxxxxxxx
Phone: 518-442-4578
_____________________________________________

________________________________________
From: Julien Chastang <chastang@xxxxxxxx>
Sent: Monday, January 6, 2014 5:40 PM
To: <idv-steering@xxxxxxxxxxxxxxxx>; idvdevelopers@xxxxxxxxxxxxxxxx
Subject: Re: [idvdevelopers] 20131220: New test release

Don,

On 1/6/14 1:55 PM, Don Murray (NOAA Affiliate) wrote:
Julien-

On 1/6/14 1:07 PM, Julien Chastang wrote:
Ideally, what we really want is something that predicts if we are going
to reach IDV memory limitations ahead of time.
In the past, the IDV didn't predict, but it did monitor and took steps
to alleviate when memory was reaching close to the limit. It also took
action to clean up memory proactively, like when all displays and data
were removed. A lot of thought was put into that. It wasn't perfect,
but seemed to work reasonably well. Apparently, the new paradigm of
letting the JVM sort this out for itself isn't working in Kevin's case.
Garbage collection algorithms have evolved significantly (or have been
replaced entirely) since the code in question was originally written.
Again, obtaining the bundle will allow us to obtain accurate metrics on
IDV memory consumption.

Yuan also suggested being aware of what other processes/applications are
running on your machine. You may have better results if you actually
*lower*  the amount of memory used via the Edit, Preference, System Tab.
The JVM will use all the memory you throw at it (and these days we are
throwing a lot, 80% of available), but if you have other apps running,
you could force page in/outs which could cause performance degradation
and even kernel panics. In fact, from looking at several bundles we have
anecdotal evidence suggesting you should never need more than ~3GB
allocated to the IDV. Making changes in the memory allocation heuristic
to not go beyond a certain hard limit (e.g., 3GB - 4GB) is something we
are studying.

Lastly, all of this will evolve as the JVM itself evolves. There will be
major improvement in Hotspot Java 8 including no more permgen in
addition to other important changes.

-Julien

Actually, this issue segues into a long-standing IDV steering committee
discussion. If possible, please pass along resource instense bundles to
the IDV team. We can put them through an analysis tool called a profiler
to gain a more accurate idea of the source of the problem.
Hopefully, Kevin can share his isl script with you to test.

Yuan's forthcoming progressive resolution feature could also help here
as it is smarter about using limited computational resources.
I doubt that's going to help Kevin who has probably already set up his
bundle to subset over the area.

Don

On 1/6/14 10:42 AM, Tyle, Kevin R wrote:
Yes, at least in the ISL script, while monitoring memory use via top,
the memory usage matches what the IDV shows as current memory usage.
Once this number meets the -Xmx limit, my ISL script grinds to a halt
(i.e., goes from ~ 4 sec per frame to over a minute, and then
eventually freezes).

_____________________________________________
Kevin Tyle, Systems Administrator
Dept. of Atmospheric & Environmental Sciences
University at Albany
Earth Science 235, 1400 Washington Avenue
Albany, NY 12222
Email: ktyle@xxxxxxxxxx
Phone: 518-442-4578
_____________________________________________


-----Original Message-----
From: Julien Chastang [mailto:chastang@xxxxxxxx]
Sent: Monday, January 06, 2014 12:40 PM
To: Tyle, Kevin R; Don.Murray@xxxxxxxx; Yuan Ho
Cc: Bob Carp; Joleen Feltz; <idv-steering@xxxxxxxxxxxxxxxx>;
idvdevelopers@xxxxxxxxxxxxxxxx
Subject: Re: [idvdevelopers] 20131220: New test release

Happy New Year All.

Kevin,

Focusing on those diagnostic numbers can be somewhat misleading
(because the internal JVM memory management algorithms are changing
and improving all the time). For example, just because you see a high
current memory number does not mean there is a problem. It could just
be that the JVM has decided not to employ one of several memory
reclamation (garbage
collection) strategies for whatever reason.

The real question is are you seeing your IDV crashing? Is there a
problem?

Best,

-Julien


On 1/6/14 10:26 AM, Tyle, Kevin R wrote:
It's a definite issue, and it becomes a real problem when running an
ISL script that loops over all frames.  I was hitting the 6GB limit
by frame 33 or so on one of my higher-RAM systems when running an ISL
script that creates US forecast radar plots from the HRRR over all 61
frames.

_____________________________________________
Kevin Tyle, Systems Administrator
Dept. of Atmospheric & Environmental Sciences University at Albany
Earth Science 235, 1400 Washington Avenue Albany, NY 12222
Email: ktyle@xxxxxxxxxx
Phone: 518-442-4578
_____________________________________________


-----Original Message-----
From: Don Murray (NOAA Affiliate) [mailto:don.murray@xxxxxxxx]
Sent: Monday, January 06, 2014 12:24 PM
To: Tyle, Kevin R; Yuan Ho
Cc: <idv-steering@xxxxxxxxxxxxxxxx>; Bob Carp; Joleen Feltz;
idvdevelopers@xxxxxxxxxxxxxxxx
Subject: Re: [idvdevelopers] 20131220: New test release

Kevin-

I believe that the IDV used to do a garbage collection (i.e. clean up
memory) after the Remove All Displays and Data action, but I know
Julien and Sean did some work to turn off all garbage collection and
let the system clean up by itself.  Ideally, it should do that, but
maybe that's not working as it should.

I agree that it was nice to have the "how much you've used" to get an
idea of when to clear the cache or whether you can actually load in
that next display.

Don

On 1/6/14 8:05 AM, Tyle, Kevin R wrote:
I've noticed that this test release does not free memory when I
select "Remove All Displays and Data".  Some times, but not most
times, memory will get freed when I proceed to load in a different
data source.  Usually I have to completely restart the IDV to get
memory use down.

I've also noticed that the three memory use values in the lower left
of the screen (i.e., the ones that appear when you toggle off the
realtime clock) have been replaced by only two values (current
use/upper limit of use) ... i.e., the maximum use in the current
session value no longer appears.  That was a nice feature ...

--Kevin

_____________________________________________
Kevin Tyle, Systems Administrator
Dept. of Atmospheric & Environmental Sciences University at Albany
Earth Science 235, 1400 Washington Avenue Albany, NY 12222
Email: ktyle@xxxxxxxxxx
Phone: 518-442-4578
_____________________________________________


-----Original Message-----
From: idvdevelopers-bounces@xxxxxxxxxxxxxxxx
[mailto:idvdevelopers-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Don
Murray
Sent: Friday, December 20, 2013 6:24 PM
To: Yuan Ho
Cc: <idv-steering@xxxxxxxxxxxxxxxx>; Bob Carp; Joleen Feltz;
idvdevelopers@xxxxxxxxxxxxxxxx
Subject: Re: [idvdevelopers] 20131220: New test release

One note about this.  If you made any bundles from the previous
progressive resolution test release, they will probably no longer
work (that's why it's called a test release ;-)).  We had to make
some significant changes to simplify the underlying code and it was
not feasible to keep the old test bundles working.  You will need to
recreate them, which would be a good test of whether things work
from scratch and from the new bundles.

And, as some of you have found, you cannot necessarily use any
bundles created under the test release with the official current
release.  But please test any bundles made with the official release
(4.1) under 5.0.

Don

On 12/20/13 3:52 PM, Yuan Ho wrote:
Hi All,
           I have a new test release here for the progressive
resolution and new image chooser. Please download from the
following link:

http://www.unidata.ucar.edu/downloads/idv/test/index.jsp


The changes in the latest:

0) the release version is 5.0 for this test release
1) remove the progressive resolution checkbox from subset panel, it
is now set as the user preference and can be changed in view drop
down menu
2) the use display area is added as a property of the datasource in
the data source properties panel
3) the show mag factor is added in the sidelegend
4) most of previous bugs should be fixed.

The design should be very close to be the final. My plan is to merge
this to the nightly release after the new year, and run more test
and prepare for the AMS short course with this version of IDV.


Yuan
--
Don Murray
NOAA/ESRL/PSD and CIRES
303-497-3596
http://www.esrl.noaa.gov/psd/people/don.murray/

_______________________________________________
idvdevelopers mailing list
idvdevelopers@xxxxxxxxxxxxxxxx
For list information, to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/


--
Don Murray
NOAA/ESRL/PSD and CU-CIRES
303-497-3596
http://www.esrl.noaa.gov/psd/people/don.murray/

_______________________________________________
idvdevelopers mailing list
idvdevelopers@xxxxxxxxxxxxxxxx
For list information, to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/

<isl debug="true" loop="1" offscreen="true" sleep="60.0minutes">
     <bundle clear="true" file="${islpath}/hrrr_radar_alltimes.xidv" 
times="0,1,2,3,4,5" wait="true" width="1024" height="768"/>
        <image file="${islpath}/hrrr_radar_single.png" 
animation_index="0,1,2,3,4,5">
            <colorbar display="display" orientation="top" width="450" 
showlines="false" values="5,10,15,20,25,30,35,40,45,50,55,60,65,70,75" 
anchor="LM" place="LM,0,-55" showunit="false"/>
            <overlay text="HRRR Forecast Reflectivity (dbZ), 1 km AGL 
${time:yyyy-MM-dd HHmm z}" fontsize="16" color="black" place="LM,0,-10" 
anchor="LM"/>
        </image>
</isl>
  • 2014 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the idvdevelopers archives: