Also have you checked the chunking compared to the read?
--
Kind regards Nick Papior
On 9 May 2016 17:56, "Chris Barker - NOAA Federal" <chris.barker@xxxxxxxx>
wrote:
On May 9, 2016, at 5:50 AM, Matthew Jones <M.Jones3@xxxxxxxxxxxxxxxxx>
wrote:
I thought so too, but it seems like the python library (which obviously
uses the C library) is significantly slower than the C library according to
my tests, which is why I'm trying to profile it.
Interesting. Line_profiler should let you get down to the line level -- so
you can put your slicking code alone on a line, and at least get that.
If slicing is slow, then you're going to have to dig into the
Cython-generated C code to see where the slowdown is. I _think_ there are
some utilities in Cython that can help with that.
Do you have examples of similar code that's a lot slower in Python?
Off the top of my head, one issue is that when you take a slice of a
variable object, it has to create a new Numpy array. If you are grabbing
small pieces at a time, that could be substantial overhead.
CHB
Thanks
Matt
------------------------------
*From:* Elizabeth A. Fischer <elizabeth.fischer@xxxxxxxxxxxx>
*Sent:* 09 May 2016 13:46
*To:* Matthew Jones
*Subject:* Re: [netcdfgroup] Fw: Profiling NetCDF4-python
I would expect most of the time to be taken up in the NetCDF-C library, not
the Python code.
On Mon, May 9, 2016 at 6:35 AM, Matthew Jones <M.Jones3@xxxxxxxxxxxxxxxxx>
wrote:
> Hi
>
>
> I am trying to profile my NetCDF4-python code to see how to improve read
> performance, which I have identified as the bottleneck. I have done this
> using kcachegrind and line_profiler.
>
>
> I want to delve a bit deeper into the library to see where the performance
> is dropping. I can't work out how to use line_profiler with the @profile
> decorator in the variable class. I also can't work out how to get
> kcachegrind to delve into the library because the slicing syntax var[...]
> is not an explicit function call, and I can't get var.__getitem__() to do
> the same thing.
>
>
> Does anyone have any ideas how I could get more information from profiling
> the netcdf4 library?
>
>
> Many thanks
>
> Matt
>
>
>
> ----------------------------------------
> Matthew Jones
> PhD Student
> Atmosphere, Oceans and Climate
> Department of Meteorology,
> University of Reading
>
> Room 288, ESSC, Harry Pitt Building,
> 3 Earley Gate, Reading, RG6 6AL, UK
>
>
> https://www.linkedin.com/pub/matthew-jones/8b/b81/25a
> http://www.met.reading.ac.uk/users/users/1887
>
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not to be made public.
>
>
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not to be made public.
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not to be made public.
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
