On Mon, May 9, 2016 at 9:02 AM, Nick Papior <nickpapior@xxxxxxxxx> wrote:
> Also have you checked the chunking compared to the read?
>
good point -- you really want to make sure your C and python code are doing
the same thing with regard to chunking, etc...
If you are reading the same file, then the chunking is the same, but if you
are using a different access pattern, it could make all the difference.
-CHB
> --
>
> Kind regards Nick Papior
> On 9 May 2016 17:56, "Chris Barker - NOAA Federal" <chris.barker@xxxxxxxx>
> wrote:
>
>
>
> On May 9, 2016, at 5:50 AM, Matthew Jones <M.Jones3@xxxxxxxxxxxxxxxxx>
> wrote:
>
> I thought so too, but it seems like the python library (which obviously
> uses the C library) is significantly slower than the C library according to
> my tests, which is why I'm trying to profile it.
>
> Interesting. Line_profiler should let you get down to the line level -- so
> you can put your slicking code alone on a line, and at least get that.
>
> If slicing is slow, then you're going to have to dig into the
> Cython-generated C code to see where the slowdown is. I _think_ there are
> some utilities in Cython that can help with that.
>
> Do you have examples of similar code that's a lot slower in Python?
>
> Off the top of my head, one issue is that when you take a slice of a
> variable object, it has to create a new Numpy array. If you are grabbing
> small pieces at a time, that could be substantial overhead.
>
> CHB
>
>
>
>
>
>
>
>
> Thanks
>
> Matt
>
>
> ------------------------------
> *From:* Elizabeth A. Fischer <elizabeth.fischer@xxxxxxxxxxxx>
> *Sent:* 09 May 2016 13:46
> *To:* Matthew Jones
> *Subject:* Re: [netcdfgroup] Fw: Profiling NetCDF4-python
>
> I would expect most of the time to be taken up in the NetCDF-C library,
> not the Python code.
>
> On Mon, May 9, 2016 at 6:35 AM, Matthew Jones <M.Jones3@xxxxxxxxxxxxxxxxx>
> wrote:
>
>> Hi
>>
>>
>> I am trying to profile my NetCDF4-python code to see how to improve read
>> performance, which I have identified as the bottleneck. I have done this
>> using kcachegrind and line_profiler.
>>
>>
>> I want to delve a bit deeper into the library to see where the
>> performance is dropping. I can't work out how to use line_profiler with the
>> @profile decorator in the variable class. I also can't work out how to get
>> kcachegrind to delve into the library because the slicing syntax var[...]
>> is not an explicit function call, and I can't get var.__getitem__() to do
>> the same thing.
>>
>>
>> Does anyone have any ideas how I could get more information from
>> profiling the netcdf4 library?
>>
>>
>> Many thanks
>>
>> Matt
>>
>>
>>
>> ----------------------------------------
>> Matthew Jones
>> PhD Student
>> Atmosphere, Oceans and Climate
>> Department of Meteorology,
>> University of Reading
>>
>> Room 288, ESSC, Harry Pitt Building,
>> 3 Earley Gate, Reading, RG6 6AL, UK
>>
>>
>> https://www.linkedin.com/pub/matthew-jones/8b/b81/25a
>> http://www.met.reading.ac.uk/users/users/1887
>>
>>
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web. Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not to be made public.
>>
>>
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not to be made public.
>
>
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not to be made public.
>
>
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
>
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@xxxxxxxx
