Re: [idvusers] Point data- poor performance - anyone have experience/solutions?

Ok, I am going to add the use driver time to the CSV point dataset.

The time driver usage in the CSV point data will be a little different. We
don't have any time information at the Field Selector level, and you will
do the same procedure as before to create the display. After the display
created, go to the display control, set View > Times > Use Time Driver
Times. Then you need to Reload the data. In the same display control, File
> Reload Data. This is the design I have and will be ready for you soon.

You only need to do it one time, saving it to a bundle will make it a lot
easier.


Yuan

On Mon, Sep 29, 2014 at 3:16 PM, Brian Mapes <bmapes@xxxxxxxxxxxxxxx> wrote:

>  Hi Yuan.
>
>  Yes I guess I imagined that much.
> Even so, I’m surprised 2448 time steps takes so long.
> I guess point processing is very costly.
>
>  So then why is IDV set up to reprocess the entire point dataset, every
> time an unrelated (or spatially below) grid display’s visibility is
> toggled?
> It doesn’t re-read and re-render the grids every time one is toggled. Why
> do it for point data? Can it be disabled?
> Could that functionality be revised just by moving some function call
> outside some loop?
> (you can see I am not an object programmer).
>
>
>
>  One thing I could invest time in (with no pleasure) would be taking the
> ASCII point data and putting the values into cells in a mostly-missing grid
> dataset.
> Obviously IDV would work much faster with that, and with netCDF4 it could
> be compressed (not huge on disk as the missing values compress well).
>
>  My point values would be square instead of round, but barbs would be
> easy enough to display, right?
> It seems silly to have to do it, but it would work.
>
>  How hard (and where on the priority list) would be the job of setting up
> a Time Driver on point data?
> How many weeks, months, years from now might it be available?
> Given our commitment to work with these 70 stations this fall,
> we need to decide whether to do this silly gridding.
>
>  Thanks,
> Brian
>
>
>
>
>  On Sep 29, 2014, at 3:03 PM, Yuan Ho <yuanho@xxxxxxxx> wrote:
>
>  Brian,
>         I think I understand the performance issue here.
>         This bundle uses animation widget to set the driver time, but you
> can not set use driver time for the local point dataset. Even though there
> are only two times step in the driver time, the IDV is likely need to
> process over 2448 time steps for this point dataset and cause the
> performance issue.  I don't have better solution other than setting the
> time declutter minute to a very big value like 5000. In the future, we need
> to add the use driver time feature to local point data.
>
>
>  Yuan
>
> On Mon, Sep 29, 2014 at 11:18 AM, Brian Mapes <bmapes@xxxxxxxxxxxxxxx>
> wrote:
>
>> Hi Yuan and Kevin.
>>
>>  This confuses me further about how time is handled for point data.
>>
>>  I notice that one can find a “Use Time Driver through the menu like
>> this:
>> <PastedGraphic-8.png>
>>
>>
>>  But it is not there in the point display control window Times tab:
>> <PastedGraphic-9.png>
>>
>>
>>  Neither choice seems to solve my grid-toggling slowness problem
>> (which I find also right now on my big new work iMac, even when I make a
>> new Point Display
>> with the downloaded data rather than the RAMADDA served file).
>>
>>
>>  Thanks for your interest!
>>  Brian
>>
>>
>>
>>  On Sep 29, 2014, at 12:45 PM, Yuan Ho <yuanho@xxxxxxxx> wrote:
>>
>>  Brian,
>>         I spent some time on this point dataset locally without any other
>> gridded data and I am not seeing any performance issues. Maybe you can
>> create the bundle with only the gridded dataset(s) and I can test the
>> performance issue by adding this point dataset locally.
>>
>>         For the point data time dimension, the IDV has two approaches:
>> binning and declutter. "Binning" is not averaging, it is
>> grouping, but for one station data it doesn't make too much sense. The
>> time declutter is more like the time subset and probably fits your
>> requirement here. The time declutter minute is controllable, when it is set
>> to 60 minutes, it means only one
>> observation within 60 minutes is selected.
>>
>>        In the previous release, the time declutter minute is set to 1
>> minute and I changed it to be more dynamically calculated in
>> the nightly release. When the IDV sees the time step over 1000, it gives
>> the warning: " do you want to show them all..". If the
>> user selects no, the IDV will use 1000 time steps to calculate the time
>> declutter minute to limit the time steps within 1000. You will see 146
>> minutes for your point dataset in your case. This number can be changed in
>> the display control window.
>>
>>       Hope this information can help.
>>
>>  Yuan
>>
>>
>> On Mon, Sep 29, 2014 at 9:06 AM, Brian Mapes <bmapes@xxxxxxxxxxxxxxx>
>> wrote:
>>
>>> This seems to have gotten spam blocked or something.
>>> Trying again.
>>>
>>> On Sep 26, 2014, at 7:27 PM, Brian Mapes <bmapes@xxxxxxxxxxxxxxx<mailto:
>>> bmapes@xxxxxxxxxxxxxxx>> wrote:
>>>
>>> Hello IDV users.
>>> I am trying to put point data in the IDV, and it works, but performs
>>> incredibly slowly.
>>> Can anyone offer advice?
>>>
>>>
>>> My test example is a single-site .csv file with 2448 time values (hourly
>>> for a 100 day field campaign). A 140 kB text file.
>>> I created multiple color-shaded plan view displays of gridded values, at
>>> one Time Driver level,
>>> plus (optionally) one point display (a ground validation) from the point
>>> data file.
>>>
>>> When the point display does not exist, I can toggle the grid displays on
>>> and off instantly, no problem.
>>> When the point display exists, toggling the grid displays on and off
>>> takes whole seconds of spinning wheel and "please wait..." time.
>>> That spoils the validation exercise (evaluating the point-data color
>>> against the gridded background shadings to see the contrast).
>>>
>>> It is replicable on 2 machines, including on a new, dedicated 8GB RAM
>>> laptop with nothing else running.
>>>
>>>
>>>
>>> Time cannot be subsetted, nor the Time Driver used, for point data.
>>> (An issue for idv-steering?)
>>> There is "rebinning" of time for point data sources, which I think means
>>> some kind of time averaging,
>>> but the hourly samples are what we want to see, not time averages
>>> thereof, and why should 1000
>>> (the rebinning magic number) be a limit?
>>>
>>> Performance seems surprisingly bad for just 2448 time levels, even given
>>> that IDV somehow thinks
>>> it has to re-read and re-render 2448 filled circles and wind barbs (just
>>> to show me ONE),
>>> whenever some OTHER display is toggled on or off.
>>> I just can't imagine where the time goes.
>>> Some kind of graphics-level inefficiency?
>>> Watching the memory counter, Megs-Gigs are involved -- for a 140 kB text
>>> datafile.
>>> ??
>>>
>>>
>>> We have a network of 70 sites with hourly data for a few years.
>>> I fear IDV will be impossible to work with at this rate.
>>> And chopping up the ascii data stream into tiny files, site by site, day
>>> by day, requiring
>>> manual new display creation for each site, every time we change the Time
>>> Driver,
>>> is obviously impractical.
>>>
>>>
>>>
>>> Anyone have experience, or know any tricks?
>>>
>>> I tried this as an idv-support ticket, but maybe it is not quite a bug --
>>> maybe we just need a user trick (hopefully)?
>>>
>>> Anyone?
>>>
>>> Thanks in advance,
>>> Brian Mapes
>>>
>>> *********************************************
>>> Brian Mapes, Professor
>>> Meteorology and Physical Oceanography
>>> RSMAS, University of Miami
>>> 4600 Rickenbacker Causeway
>>> Miami, FL 33149-1031
>>>
>>> phone: (305) 421-4275
>>> fax: (305) 421-4696
>>> email: mapes@xxxxxxxxx<mailto:mapes@xxxxxxxxx>
>>> Web: http://www.rsmas.miami.edu/users/bmapes/
>>> **********************************************
>>>
>>>
>>>
>>> _______________________________________________
>>> idvusers mailing list
>>> idvusers@xxxxxxxxxxxxxxxx
>>> For list information, to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>>>
>>>
>>
>>
>
>


  • 2014 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the idvusers archives: