>
>
> This is not a problem with pqact(1) *per se*, but with inefficient
> decoders (pure Python, for example, is about three orders of magnitude
> slower than C).
>
While Python is certainly slower than C (though I strongly question 1000x
slower), I don't want anyone taking away from this that Python is a poor
choice for an LDM decoder. The entire infrastructure ingesting *all* of the
NEXRAD Level 2 feed into the Amazon S3 archive (and the realtime chunks
bucket) is powered by a single pure Python process that takes 30% of a
single CPU. We're also using Python to stitch together in realtime the
GOES-16/17 imagery tiles being sent over NOAAPORT. These are not small data
feeds.
I'll note these applications are using long-running, persistent processes
that have data continually piped in. Where Python becomes a problem is if
you are starting up lots of individual processes, say one per product when
you have 10s of products per second. In that case the startup/shutdown time
of the Python interpreter will kill you. I tried this the other day with
the NEXRAD3 feed and managed to get a machine up to a load average of 185.
:)
Ryan
--
Ryan May, Ph.D.
Software Engineer
UCAR/Unidata
Boulder, CO
