You could run apache or nginx on port 80 (either on the host or a separate
container) - using http redirects to https (port 443) - and have certbot
running in that container/host context (In case of container - make sure to
have the /etc/letsencrypt persisted, e.g. by using a -v
/host/path:/etc/letsencrypt - also do something to ensure that certbot is
triggered regularity to update your certs)
Then run the tds container mounting -v /host/path:/etc/letsencrypt:ro and
update the configuration to use the certificate presented by certbot.
ons. 31. juli 2024 kl. 01:38 skrev Jim Fluke <james.fluke@xxxxxxxxxxxxx>:
> Okay, we are trying to install an "official" certificate using Certbot.
> Apparently, using Certbot is now the required way to install certificates
> at CSU, and we have done this successfully for Apache and ngnix running on
> the host. The problem is that we are having trouble creating a Certbot
> configuration that works for the TDS running in a container (
> thredds-docker <https://github.com/Unidata/thredds-docker>). Has anyone
> done this? If so, can you share your Certbot configuration?
>
> Thanks,
> Jim
>
> On 7/12/24 03:45, Christian Skarby wrote:
>
> *** Caution: EXTERNAL Sender ***
>
> *The easiest and best is usually to get a certificate from one of the
> renowned suppliers generally included in the trust store of big browsers
> and operating systems. *
> Check out https://letsencrypt.org/ which provides certificates for free.
> If it is possible to expose the http-port (tcp/80) of your server to the
> internet, that is an easy way to start using Let's Encrypt - and if
> necessary they also provides other ways to identify ownership of hostnames,
> e.g. by providing DNS-records.
>
> If you really would like to make your own test certificates, you could
> check out https://github.com/OpenVPN/easy-rsa
> The certificates are regular SSL/TLS-certificates, and can be used for any
> protocol encrypted with TLS.
> Read through
> https://github.com/OpenVPN/easy-rsa/blob/master/doc/Intro-To-PKI.md which
> is an introduction to public key infrastructure.
> Your users must also understand the risk of giving you super powers:
> Certificate Authorities (CA) are trusted entities in your operating
> system/browser, and could issue certificates for any hostname. If they
> trust your CA, you could in theory make certificates for any existing (or
> non-existing domain name) and make their browser/application trust that
> site as you provide a valid certificate issued by one of their trusted CAs.
> Operating a CA also requires understanding of the trust model and to keep
> track of the different certificate expiry dates within the certificate
> chain from your root certificate and down to the service certificate.
>
> Again, if possible - always use certificates from official providers.
> Rolling your own CA is a big responsibility, and not for the faint of heart
>
> --
> Best Regards,
>
> Christian Skarby
> MET Norway
>
> fre. 12. juli 2024 kl. 00:04 skrev Jim Fluke <james.fluke@xxxxxxxxxxxxx>:
>
>> Pols,
>>
>> Well, by actually reading the rest of the instructions in the TDS
>> documentation
>> <https://docs.unidata.ucar.edu/tds/current/userguide/enable_tls_encryption.html>
>> I was able to set the certificateKeystorePassword, which fixed this
>> problem. At least for website access if I push though the self-signed
>> certificate warnings.
>>
>> But, pydap is failing due to the self-signed certificate and I haven't
>> found a way around it yet:
>> ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
>> certificate verify failed: self-signed certificate (_ssl.c:1000)
>>
>> If anyone knows a way around that please let me know.
>>
>> Thanks,
>> Jim
>>
>> On 7/11/24 11:47, Jim Fluke wrote:
>>
>> Pols,
>>
>> I created a self-signed certificate since it's just for testing right
>> now. So far I can't get it to work though. Here are the errors I get at TDS
>> start up:
>> 10-Jul-2024 15:26:16.372 SEVERE [main]
>> org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to
>> initialize component [Connector["https-openssl-nio-8443"]]
>> org.apache.catalina.LifecycleException: Protocol handler
>> initialization failed
>> .
>> .
>> .
>> Caused by: java.lang.IllegalArgumentException: Keystore was
>> tampered with, or password was incorrect
>> .
>> .
>> .
>> Caused by: java.security.UnrecoverableKeyException: Password
>> verification failed
>>
>> And, I am still using 8443. Also because this is a test environment.
>>
>> Do you have any idea where I can change the password. If that really is
>> the problem.
>>
>> Thanks,
>> Jim
>>
>> On 7/10/24 01:33, Pols, Maarten wrote:
>>
>> *** Caution: EXTERNAL Sender ***
>>
>> Dear Jim,
>>
>>
>>
>> I think you are right, first setup a SSL certificate, I’m also using the
>> thredds docker image, together with a nginx proxy server.
>>
>>
>>
>> *M.J. (Maarten) Pols*
>> *Producten en services*
>> *Systeem- en applicatiebeheerder*
>>
>>
>>
>> Botter 11-29, 8232 JN Lelystad (tevens postadres)
>> Berkenweg 7, Amersfoort | Informaticalaan 8, Delft
>>
>> Telefoon 0320 294292
>> Internet *www.hkv.nl <http://www.hkv.nl/>*
>>
>>
>>
>> HKV, de kennisondernemer voor water en veiligheid
>>
>> *Van:* Jim Fluke <james.fluke@xxxxxxxxxxxxx> <james.fluke@xxxxxxxxxxxxx>
>> *Verzonden:* Tuesday, 9 July 2024 19:45
>> *Aan:* Pols, Maarten <M.Pols@xxxxxx> <M.Pols@xxxxxx>;
>> thredds@xxxxxxxxxxxxxxxx
>> *Onderwerp:* Re: [thredds] Authentication problems with the TDS and pydap
>>
>>
>>
>> ## Let op: deze mail is afkomstig van een externe afzender. Meer
>> informatie over waarom dit belangrijk is
>> <https://aka.ms/LearnAboutSenderIdentification>
>>
>>
>>
>> Pols,
>>
>> Thank you for your response!
>>
>> But, it still does not work. I think I probably need this, or something
>> like it, but it's not enough.
>>
>> Now the web browser authentication fails with this message:
>> Secure Connection Failed
>>
>> An error occurred during a connection to localhost. PR_END_OF_FILE_ERROR
>>
>> Error code: PR_END_OF_FILE_ERROR
>>
>> The page you are trying to view cannot be shown because the
>> authenticity of the received data could not be verified.
>> Please contact the website owners to inform them of this problem.
>>
>> And the pydap authentication fails with this message:
>> ssl.SSLEOFError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in
>> violation of protocol (_ssl.c:1000)
>>
>> Which seems to indicate that I need to add an SSL certificate, which I
>> have not done. Again, I am using the thredds-docker image, which does not
>> have a certificate by default. And the port forwarding that it does might
>> be an issue as well.
>>
>> I'll try the certificate, but other suggestions would be very welcome.
>>
>> Jim
>>
>> On 7/9/24 00:35, Pols, Maarten wrote:
>>
>> *** Caution: EXTERNAL Sender ***
>>
>> Dear Jim,
>>
>>
>>
>> This problem cost me months to cover. It was working in previous versions
>> of thredds but after een upgrade it broke my python scripts.
>>
>> First of all, don’t upgrade to the latest numpy packages, it will break
>> pydap, latest working version is 1.26.x
>>
>>
>>
>> Than to solve this issue, you need to change applicationContext.xml file,
>> this file is in webapps -> thredds -> WEB-INF
>>
>> You need to change line 112 and 113:
>>
>>
>>
>> * <bean id="restrictedDatasetAuthorizer"
>> class="thredds.servlet.restrict.TomcatAuthorizer">*
>>
>> * <property name="useSSL" value="false"/>*
>>
>> * <property name="sslPort" value="8443"/>*
>>
>> * </bean>*
>>
>>
>>
>> Into
>>
>>
>>
>> * <bean id="restrictedDatasetAuthorizer"
>> class="thredds.servlet.restrict.TomcatAuthorizer">*
>>
>> * <property name="useSSL" value="true"/>*
>>
>> * <property name="sslPort" value="443"/>*
>>
>> * </bean>*
>>
>>
>>
>> This was solving the issue in my case, and I hope it will help you.
>>
>>
>>
>>
>>
>> *M.J. (Maarten) Pols Products and Services System and application
>> administrator *
>>
>>
>>
>> Botter 11-29, 8232 JN Lelystad, The Netherlands (also postal address)
>> Berkenweg 7, Amersfoort | Informaticalaan 8, Delft
>>
>> Telephone +31 (0)320 294292
>> Internet *www.hkv.nl/en/ <http://www.hkv.nl/en/>*
>>
>>
>>
>> HKV, knowledge entrepreneurs in flood risk and water resources management
>>
>> *Van:* thredds <thredds-bounces@xxxxxxxxxxxxxxxx>
>> <thredds-bounces@xxxxxxxxxxxxxxxx> *Namens *Jim Fluke
>> *Verzonden:* Tuesday, 9 July 2024 00:04
>> *Aan:* thredds@xxxxxxxxxxxxxxxx
>> *Onderwerp:* [thredds] Authentication problems with the TDS and pydap
>>
>>
>>
>> ## Let op: deze mail is afkomstig van een externe afzender. Meer
>> informatie over waarom dit belangrijk is
>> <https://aka.ms/LearnAboutSenderIdentification>
>>
>>
>>
>>
>> Hello,
>>
>> I'm now trying to get user authentication working with our thredds-docker
>> based TDS. I'm pretty sure I have the configuration set up to enable
>> authentication as described in the TDS manual's "Restrict Access To The
>> TDS
>> <https://docs.unidata.ucar.edu/tds/current/userguide/restict_access_to_tds.html#restrict-access-by-dataset-in-tds-catalogs>"
>> page. And I have verified this by accessing the TDS from a browser and
>> having the credentials entry pop-up window display and work correctly.
>>
>> But, I can't get the authentication to work in Python with pydap.
>> According to the pydap documentation the credentials should be added to the
>> URL this way:
>>
>> >>> from pydap.client import open_url
>> >>> dataset = open_url('
>> http://username:password@xxxxxxxxxxxxxxxxxx/path/to/dataset')
>>
>> But because Digested Passwords
>> <https://docs.unidata.ucar.edu/tds/current/userguide/digested_passwords.html>
>> are enabled for our TDS, it seems clear that I should use the digested
>> password, so this is what I tried:
>>
>> >>> from pydap.client import open_url
>> >>> dataset = open_url('
>> http://fluke:d1ef3ce7e7c41de74192a362524ad0a460692a222d9dd796ee383b56e446d749
>> $1$d03ce0f88475505a68bd0eb37fa570df8120e59ccf62a4f580a55ad612f695c0e385893fe7205f7c181b221ab49bc817d4a33a2b
>>
>> 2bb727fdc0ee3420e7e5b99e@localhost:7000/thredds/dodsC/cloudsat-data/2B-GEOPROF.P1_R05/2008/366/2008366031107_14239_CS_2B-GEOPROF_GRANULE_P1_R05_E02_F00.hdf
>> ')
>>
>> But it does not work. Here is the output:
>>
>> @ ~/devRepos/thredds-dpc-gh-actual/tests$ docker-compose run --rm
>> test_opendap
>> url:
>> http://fluke:d1ef3ce7e7c41de74192a362524ad0a460692a222d9dd796ee383b56e446d749$1$d03ce0f88475505a68bd0eb37fa570df8120e59ccf62a4f580a55ad612f695c0e385893fe7205f7c181b221ab49bc817d4a33a2b
>>
>> 2bb727fdc0ee3420e7e5b99e@localhost:7000/thredds/dodsC/cloudsat-data/2B-GEOPROF.P1_R05/2008/366/2008366031107_14239_CS_2B-GEOPROF_GRANULE_P1_R05_E02_F00.hdf
>>
>> Traceback (most recent call last):
>> File "/app/opendap_pydap.py", line 8, in <module>
>> dataset = open_url(url)
>> ^^^^^^^^^^^^^
>> File "/opt/conda/lib/python3.12/site-packages/pydap/client.py", line 68,
>> in open_url
>> handler = pydap.handlers.dap.DAPHandler(url, application, session,
>> output_grid,
>>
>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>
>> File "/opt/conda/lib/python3.12/site-packages/pydap/handlers/dap.py",
>> line 71, in __init__
>> self.make_dataset()
>> File "/opt/conda/lib/python3.12/site-packages/pydap/handlers/dap.py",
>> line 96, in make_dataset
>> self.dataset_from_dap2()
>> File "/opt/conda/lib/python3.12/site-packages/pydap/handlers/dap.py",
>> line 109, in dataset_from_dap2
>> pydap.net.raise_for_status(r)
>> File "/opt/conda/lib/python3.12/site-packages/pydap/net.py", line 38, in
>> raise_for_status
>> raise HTTPError(
>> webob.exc.HTTPError: 401 Unauthorized
>> <!doctype html><html lang="en"><head><title>HTTP Status 401 –
>> Unauthorized</title><style type="text/css">body
>> {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b
>> {color:white;background-co
>> lor:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3
>> {font-size:14px;} p {font-size:12px;} a {color:black;} .line
>> {height:1px;background-color:#525D76;border:none;}</style></head><bod
>> y><h1>HTTP Status 401 – Unauthorized</h1><hr class="line"
>> /><p><b>Type</b> Status Report</p><p><b>Description</b> The request has not
>> been applied to the target resource because it lacks va
>> lid authentication credentials for that resource.</p><hr class="line"
>> /><h3>Apache Tomcat</h3></body></html>
>>
>> So, am I right to be using the digested password? Do you see anything
>> else that could be wrong? Why does this work for the browser but not for
>> pydap?
>>
>> I will add that the algorithm for the CredentialHandler is "sha-*512*"
>> in the ~tomcat/conf/server.xml file inside the container, so that is why
>> the digested password is an sha512 digest. And the clear text password is
>> "flukeTmp". I'll be changing that for our production system.
>>
>> And, all of this - the TDS configuration and the test python script with
>> the above URL - are now checked in to our thredds-dpc
>> <https://github.com/JimFluke/thredds-dpc/tree/master> repository on
>> GitHub so you can look at the details there.
>>
>> Any help would be greatly appreciated.
>>
>> Thanks,
>> Jim
>>
>>
>>
>>
>>
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web. Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> https://www.unidata.ucar.edu/mailing_lists/
>>
>
>
