Re: [ldm-users] 20200423: Re: Efficiency of splitting pqacts

  • To: Steven Emmerson <emmerson@xxxxxxxx>, Mohan Ramamurthy <mohan@xxxxxxxx>, Tom Yoksas <yoksas@xxxxxxxx>
  • Subject: Re: [ldm-users] 20200423: Re: Efficiency of splitting pqacts
  • From: "Knight, David" <dknight@xxxxxxxxxx>
  • Date: Fri, 24 Apr 2020 00:55:07 +0000
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=albany.edu; dmarc=pass action=none header.from=albany.edu; dkim=pass header.d=albany.edu; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Vbv0k11oiuZQWKbC8nMPJJEmhQctBrQiXeb2yQheocA=; b=bpo+UlCIwEoeFS+nsxJI7LyTClR7lWgw1br45x1XISuV0PEazcmOIorn6YAPlWSJZQkIDu/pZS7yijhkDSZpoZlGr6K8lSPorU+CoZDmBTR90UTBWbJ2ysN0cJAiDV6Dr9vhKgJSbnU1CziluuQbAv0W/ZwglmcaMEhuq+ucRLUxEqK0hFgxekU5xXlYIwijaevH76VQsqy/KPqcnMbGlR65MSAKqdxJhXfRHlNgrcnXqBW2UYBSxx5ecdnSI3ngHRMqAUJ381/5RUulI6nauAZhRhjCUbxRXnRrPYMtlQSPs+fYxBnrx/lz5SMEtigl6prPjqwYc1EGIQcK4RFi9w==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZLgPR1A/3lt6AWhxpLl8aFAunFKtJeGGbZLWTTWcFIjeBoXJTwsY+ZoN+foFvkjqhv5U4PAf7Oyvm24EAPMdcfaoNHxzVudV/k3RjLjeLAnrdu1/nS3H9Bnmfyq/Qll2mBJC+p0LMz1NTKAReXW4KgGnT9p4Swylh3tE1q828B9AHd5qypgPfdh1E+xFOXOsqO84D+4qdsPEVxCBi2JeuKkf0Czrzcb7xAb3mng10h5h57EcGbeBwM9JzaF8CYGnCSko5MFpv8MNK/UHv3UH7OSWFKcnzl72AnK7twOjWYx8r5CJItSdqcWUKzVlN22yyDP0O9d5eGPXsJkLcsvHpg==
  • Authentication-results: spf=none (sender IP is ) smtp.mailfrom=dknight@xxxxxxxxxx;
I have two concerns. Which you can feel free to ignore if you want.

1: (pure Python, for example, is about three orders of magnitude slower than C).

Three orders of magnitude is a lot!

2: I have retired. I suspect some people at Unidata might want to also retire. 
One day.

I remember when Java was the new thing. Then Python. I can’t remember how many 
new things that have been tried then abandoned.

My concern is that Unidata is always chasing the new thing.
My concern is that you have key people who could retire at any time.

I’m not sure how you address these things. I assume you have a plan.

David
> 

> On Apr 23, 2020, at 7:51 PM, Steven Emmerson <emmerson@xxxxxxxx> wrote:
> 
> Hi Daryl,
> 
> On Thu, Apr 23, 2020 at 1:37 PM Herzmann, Daryl E [AGRON] 
> <akrherz@xxxxxxxxxxx> wrote:
> I am sure Unidata will correct my ignorance / incorrect details, but my 
> understanding is that an individual pqact process can only do 32 "things" at 
> one time, or there's 32 slots available for work.
> 
> The "things" are file descriptors and the Unix standard only guarantees 32 of 
> them. Modern O/S provide much more.
>  
> Now, the above depends on the action.  If you run `PIPE -close`, the slot
> 
> The "slot" is a file descriptor.
>  
> can be used for another product even with the PIPEd process still running...
> 
> The PIPE-d process might still be running, but the file descriptor will have 
> been closed -- so, yes, that descriptor can be reused.
>  
>   This type of action can lead LDM to DOSing the server it is on as it will 
> fire off as many PIPE'd processes that it can.
> 
> If the decoders take a long time to process the data-product, then, yes, you 
> can wind up with many of them. This is not a problem with pqact(1) per se, 
> but with inefficient decoders (pure Python, for example, is about three 
> orders of magnitude slower than C).
> 
> Regards,
> Steve Emmerson
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
> 
> 
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit: 
> https://www.unidata.ucar.edu/mailing_lists/ 

David Knight
11 Mercer St.
Albany, NY 12203
(518)-438-9269 (leave message)
dknight@xxxxxxxxxx



  • 2020 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: