Hello Jim,
This is a new question - as your first was how to handle certificates. I
suppose that your setup is like in the attached image. I would recommend to
use a firewall (either external and/or a builtin firewall in the operating
system only allow new connections to desired ports, possibly ssh from a
jump host in addition to tcp/80 and tcp/443 from the world - or selected ip
ranges if your users are not the general public.)
[image: tds-letsencrypt.png]
In order for thredds data server (TDS) to craft correct URLs tomcat and
thredds would need to know the protocol (https) and hostname that the user
is connecting to. As nginx is a reverse proxy, both nginx and tomcat must
be configured accordingly.
It has been a while since I configured tomcat and TDS, but I believe that
you could use the following for your tomcat connector configuration.
<Connector port="7000" protocol="HTTP/1.1"
maxThreads="5000" scheme="https" secure="true"
relaxedQueryChars="<>[\]{|}"
clientAuth="false" />
As for nginx you might need something like the following before the
proxy_pass directive:
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $http_host;
I can't remember if there are more things that needs to be configured, or
if TDS will recognize these http headers and craft the URLs correctly.
However I hope these pointers will get you in the right direction :)
Good luck!
--
Best regards,
Christian Skarby
MET Norway, IT
tir. 8. okt. 2024 kl. 20:38 skrev Jim Fluke <james.fluke@xxxxxxxxxxxxx>:
> Christian,
>
> I'm now trying to use an nginx as a proxy - or maybe I should say a
> reverse proxy - but I'm still having trouble. We can't publicly expose a
> server here that is http, it has to be https, so I'm trying to do things
> differently than what you describe. I have nginx configured to use https
> and to forward requests to the TDS using http. Here is the server block in
> nginx.conf:
>
> server {
> listen 443 ssl;
>
> server_name gcin01.cira.colostate.edu;
> ssl_certificate
> /root/CERTS/JUL2024/gcin01_cira_colostate_edu_cert.cer-CertOnlyPEMEncoded;
> ssl_certificate_key /root/CERTS/JUL2024/gcin01.key;
>
> location / {
> proxy_pass http://localhost:7000/;
> }
> }
>
> The certificate settings work fine and give us an https connection to our
> TDS website, but only for the catalog pages for navigating the dataset.
> When I select the OpenDAP service button it gives me a localhost:7000
> DataURL for the file I'm accessing. Example:
>
> http://localhost:7000/thredds/dodsC/cloudsat-data/2B-GEOPROF.P1_R05/2013/180/2013180111833_38146_CS_2B-GEOPROF_GRANULE_P1_R05_E06_F00.hdf
>
> If I manually change this to start with https://gcin01 then it works
> fine. Is there a way to configure the OpenDAP service to use the https
> start to the URL?
>
> More importantly, I can't get pydap to work through the nginx server. When
> I give it a URL like this:
>
> https://gcin01/thredds/dodsC/cloudsat-data/2B-GEOPROF.P1_R05/2013/180/2013180111833_38146_CS_2B-GEOPROF_GRANULE_P1_R05_E06_F00.hdf
>
> It raises and exception ending with:
> ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate
> verify failed: unable to get local issuer certificate (_ssl.c:1006)
>
> Any suggestions on how to fix this would be greatly appreciated.
>
> Thanks,
> Jim
>
> On 7/31/24 14:49, Christian Skarby wrote:
>
> *** Caution: EXTERNAL Sender ***
> You could run apache or nginx on port 80 (either on the host or a separate
> container) - using http redirects to https (port 443) - and have certbot
> running in that container/host context (In case of container - make sure to
> have the /etc/letsencrypt persisted, e.g. by using a -v
> /host/path:/etc/letsencrypt - also do something to ensure that certbot is
> triggered regularity to update your certs)
>
> Then run the tds container mounting -v /host/path:/etc/letsencrypt:ro and
> update the configuration to use the certificate presented by certbot.
>
> ons. 31. juli 2024 kl. 01:38 skrev Jim Fluke <james.fluke@xxxxxxxxxxxxx>:
>
>> Okay, we are trying to install an "official" certificate using Certbot.
>> Apparently, using Certbot is now the required way to install certificates
>> at CSU, and we have done this successfully for Apache and ngnix running on
>> the host. The problem is that we are having trouble creating a Certbot
>> configuration that works for the TDS running in a container (
>> thredds-docker <https://github.com/Unidata/thredds-docker>). Has anyone
>> done this? If so, can you share your Certbot configuration?
>>
>> Thanks,
>> Jim
>>
>> On 7/12/24 03:45, Christian Skarby wrote:
>>
>> *** Caution: EXTERNAL Sender ***
>>
>> *The easiest and best is usually to get a certificate from one of the
>> renowned suppliers generally included in the trust store of big browsers
>> and operating systems. *
>> Check out https://letsencrypt.org/ which provides certificates for free.
>> If it is possible to expose the http-port (tcp/80) of your server to the
>> internet, that is an easy way to start using Let's Encrypt - and if
>> necessary they also provides other ways to identify ownership of hostnames,
>> e.g. by providing DNS-records.
>>
>> If you really would like to make your own test certificates, you could
>> check out https://github.com/OpenVPN/easy-rsa
>> The certificates are regular SSL/TLS-certificates, and can be used for
>> any protocol encrypted with TLS.
>> Read through
>> https://github.com/OpenVPN/easy-rsa/blob/master/doc/Intro-To-PKI.md
>> which is an introduction to public key infrastructure.
>> Your users must also understand the risk of giving you super powers:
>> Certificate Authorities (CA) are trusted entities in your operating
>> system/browser, and could issue certificates for any hostname. If they
>> trust your CA, you could in theory make certificates for any existing (or
>> non-existing domain name) and make their browser/application trust that
>> site as you provide a valid certificate issued by one of their trusted CAs.
>> Operating a CA also requires understanding of the trust model and to keep
>> track of the different certificate expiry dates within the certificate
>> chain from your root certificate and down to the service certificate.
>>
>> Again, if possible - always use certificates from official providers.
>> Rolling your own CA is a big responsibility, and not for the faint of heart
>>
>> --
>> Best Regards,
>>
>> Christian Skarby
>> MET Norway
>>
>> fre. 12. juli 2024 kl. 00:04 skrev Jim Fluke <james.fluke@xxxxxxxxxxxxx>:
>>
>>> Pols,
>>>
>>> Well, by actually reading the rest of the instructions in the TDS
>>> documentation
>>> <https://docs.unidata.ucar.edu/tds/current/userguide/enable_tls_encryption.html>
>>> I was able to set the certificateKeystorePassword, which fixed this
>>> problem. At least for website access if I push though the self-signed
>>> certificate warnings.
>>>
>>> But, pydap is failing due to the self-signed certificate and I haven't
>>> found a way around it yet:
>>> ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
>>> certificate verify failed: self-signed certificate (_ssl.c:1000)
>>>
>>> If anyone knows a way around that please let me know.
>>>
>>> Thanks,
>>> Jim
>>>
>>> On 7/11/24 11:47, Jim Fluke wrote:
>>>
>>> Pols,
>>>
>>> I created a self-signed certificate since it's just for testing right
>>> now. So far I can't get it to work though. Here are the errors I get at TDS
>>> start up:
>>> 10-Jul-2024 15:26:16.372 SEVERE [main]
>>> org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to
>>> initialize component [Connector["https-openssl-nio-8443"]]
>>> org.apache.catalina.LifecycleException: Protocol handler
>>> initialization failed
>>> .
>>> .
>>> .
>>> Caused by: java.lang.IllegalArgumentException: Keystore was
>>> tampered with, or password was incorrect
>>> .
>>> .
>>> .
>>> Caused by: java.security.UnrecoverableKeyException: Password
>>> verification failed
>>>
>>> And, I am still using 8443. Also because this is a test environment.
>>>
>>> Do you have any idea where I can change the password. If that really is
>>> the problem.
>>>
>>> Thanks,
>>> Jim
>>>
>>> On 7/10/24 01:33, Pols, Maarten wrote:
>>>
>>> *** Caution: EXTERNAL Sender ***
>>>
>>> Dear Jim,
>>>
>>>
>>>
>>> I think you are right, first setup a SSL certificate, I’m also using the
>>> thredds docker image, together with a nginx proxy server.
>>>
>>>
>>>
>>> *M.J. (Maarten) Pols*
>>> *Producten en services*
>>> *Systeem- en applicatiebeheerder*
>>>
>>>
>>>
>>> Botter 11-29, 8232 JN Lelystad (tevens postadres)
>>> Berkenweg 7, Amersfoort | Informaticalaan 8, Delft
>>>
>>> Telefoon 0320 294292
>>> Internet *www.hkv.nl <http://www.hkv.nl/>*
>>>
>>>
>>>
>>> HKV, de kennisondernemer voor water en veiligheid
>>>
>>> *Van:* Jim Fluke <james.fluke@xxxxxxxxxxxxx> <james.fluke@xxxxxxxxxxxxx>
>>> *Verzonden:* Tuesday, 9 July 2024 19:45
>>> *Aan:* Pols, Maarten <M.Pols@xxxxxx> <M.Pols@xxxxxx>;
>>> thredds@xxxxxxxxxxxxxxxx
>>> *Onderwerp:* Re: [thredds] Authentication problems with the TDS and
>>> pydap
>>>
>>>
>>>
>>> ## Let op: deze mail is afkomstig van een externe afzender. Meer
>>> informatie over waarom dit belangrijk is
>>> <https://aka.ms/LearnAboutSenderIdentification>
>>>
>>>
>>>
>>> Pols,
>>>
>>> Thank you for your response!
>>>
>>> But, it still does not work. I think I probably need this, or something
>>> like it, but it's not enough.
>>>
>>> Now the web browser authentication fails with this message:
>>> Secure Connection Failed
>>>
>>> An error occurred during a connection to localhost. PR_END_OF_FILE_ERROR
>>>
>>> Error code: PR_END_OF_FILE_ERROR
>>>
>>> The page you are trying to view cannot be shown because the
>>> authenticity of the received data could not be verified.
>>> Please contact the website owners to inform them of this problem.
>>>
>>> And the pydap authentication fails with this message:
>>> ssl.SSLEOFError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in
>>> violation of protocol (_ssl.c:1000)
>>>
>>> Which seems to indicate that I need to add an SSL certificate, which I
>>> have not done. Again, I am using the thredds-docker image, which does not
>>> have a certificate by default. And the port forwarding that it does might
>>> be an issue as well.
>>>
>>> I'll try the certificate, but other suggestions would be very welcome.
>>>
>>> Jim
>>>
>>> On 7/9/24 00:35, Pols, Maarten wrote:
>>>
>>> *** Caution: EXTERNAL Sender ***
>>>
>>> Dear Jim,
>>>
>>>
>>>
>>> This problem cost me months to cover. It was working in previous
>>> versions of thredds but after een upgrade it broke my python scripts.
>>>
>>> First of all, don’t upgrade to the latest numpy packages, it will break
>>> pydap, latest working version is 1.26.x
>>>
>>>
>>>
>>> Than to solve this issue, you need to change applicationContext.xml
>>> file, this file is in webapps -> thredds -> WEB-INF
>>>
>>> You need to change line 112 and 113:
>>>
>>>
>>>
>>> * <bean id="restrictedDatasetAuthorizer"
>>> class="thredds.servlet.restrict.TomcatAuthorizer">*
>>>
>>> * <property name="useSSL" value="false"/>*
>>>
>>> * <property name="sslPort" value="8443"/>*
>>>
>>> * </bean>*
>>>
>>>
>>>
>>> Into
>>>
>>>
>>>
>>> * <bean id="restrictedDatasetAuthorizer"
>>> class="thredds.servlet.restrict.TomcatAuthorizer">*
>>>
>>> * <property name="useSSL" value="true"/>*
>>>
>>> * <property name="sslPort" value="443"/>*
>>>
>>> * </bean>*
>>>
>>>
>>>
>>> This was solving the issue in my case, and I hope it will help you.
>>>
>>>
>>>
>>>
>>>
>>> *M.J. (Maarten) Pols Products and Services System and application
>>> administrator *
>>>
>>>
>>>
>>> Botter 11-29, 8232 JN Lelystad, The Netherlands (also postal address)
>>> Berkenweg 7, Amersfoort | Informaticalaan 8, Delft
>>>
>>> Telephone +31 (0)320 294292
>>> Internet *www.hkv.nl/en/ <http://www.hkv.nl/en/>*
>>>
>>>
>>>
>>> HKV, knowledge entrepreneurs in flood risk and water resources
>>> management
>>>
>>> *Van:* thredds <thredds-bounces@xxxxxxxxxxxxxxxx>
>>> <thredds-bounces@xxxxxxxxxxxxxxxx> *Namens *Jim Fluke
>>> *Verzonden:* Tuesday, 9 July 2024 00:04
>>> *Aan:* thredds@xxxxxxxxxxxxxxxx
>>> *Onderwerp:* [thredds] Authentication problems with the TDS and pydap
>>>
>>>
>>>
>>> ## Let op: deze mail is afkomstig van een externe afzender. Meer
>>> informatie over waarom dit belangrijk is
>>> <https://aka.ms/LearnAboutSenderIdentification>
>>>
>>>
>>>
>>>
>>> Hello,
>>>
>>> I'm now trying to get user authentication working with our
>>> thredds-docker based TDS. I'm pretty sure I have the configuration set up
>>> to enable authentication as described in the TDS manual's "Restrict
>>> Access To The TDS
>>> <https://docs.unidata.ucar.edu/tds/current/userguide/restict_access_to_tds.html#restrict-access-by-dataset-in-tds-catalogs>"
>>> page. And I have verified this by accessing the TDS from a browser and
>>> having the credentials entry pop-up window display and work correctly.
>>>
>>> But, I can't get the authentication to work in Python with pydap.
>>> According to the pydap documentation the credentials should be added to the
>>> URL this way:
>>>
>>> >>> from pydap.client import open_url
>>> >>> dataset = open_url('
>>> http://username:password@xxxxxxxxxxxxxxxxxx/path/to/dataset')
>>>
>>> But because Digested Passwords
>>> <https://docs.unidata.ucar.edu/tds/current/userguide/digested_passwords.html>
>>> are enabled for our TDS, it seems clear that I should use the digested
>>> password, so this is what I tried:
>>>
>>> >>> from pydap.client import open_url
>>> >>> dataset = open_url('
>>> http://fluke:d1ef3ce7e7c41de74192a362524ad0a460692a222d9dd796ee383b56e446d749
>>> $1$d03ce0f88475505a68bd0eb37fa570df8120e59ccf62a4f580a55ad612f695c0e385893fe7205f7c181b221ab49bc817d4a33a2b
>>>
>>> 2bb727fdc0ee3420e7e5b99e@localhost:7000/thredds/dodsC/cloudsat-data/2B-GEOPROF.P1_R05/2008/366/2008366031107_14239_CS_2B-GEOPROF_GRANULE_P1_R05_E02_F00.hdf
>>> ')
>>>
>>> But it does not work. Here is the output:
>>>
>>> @ ~/devRepos/thredds-dpc-gh-actual/tests$ docker-compose run --rm
>>> test_opendap
>>> url:
>>> http://fluke:d1ef3ce7e7c41de74192a362524ad0a460692a222d9dd796ee383b56e446d749$1$d03ce0f88475505a68bd0eb37fa570df8120e59ccf62a4f580a55ad612f695c0e385893fe7205f7c181b221ab49bc817d4a33a2b
>>>
>>> 2bb727fdc0ee3420e7e5b99e@localhost:7000/thredds/dodsC/cloudsat-data/2B-GEOPROF.P1_R05/2008/366/2008366031107_14239_CS_2B-GEOPROF_GRANULE_P1_R05_E02_F00.hdf
>>>
>>> Traceback (most recent call last):
>>> File "/app/opendap_pydap.py", line 8, in <module>
>>> dataset = open_url(url)
>>> ^^^^^^^^^^^^^
>>> File "/opt/conda/lib/python3.12/site-packages/pydap/client.py", line
>>> 68, in open_url
>>> handler = pydap.handlers.dap.DAPHandler(url, application, session,
>>> output_grid,
>>>
>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>
>>> File "/opt/conda/lib/python3.12/site-packages/pydap/handlers/dap.py",
>>> line 71, in __init__
>>> self.make_dataset()
>>> File "/opt/conda/lib/python3.12/site-packages/pydap/handlers/dap.py",
>>> line 96, in make_dataset
>>> self.dataset_from_dap2()
>>> File "/opt/conda/lib/python3.12/site-packages/pydap/handlers/dap.py",
>>> line 109, in dataset_from_dap2
>>> pydap.net.raise_for_status(r)
>>> File "/opt/conda/lib/python3.12/site-packages/pydap/net.py", line 38,
>>> in raise_for_status
>>> raise HTTPError(
>>> webob.exc.HTTPError: 401 Unauthorized
>>> <!doctype html><html lang="en"><head><title>HTTP Status 401 –
>>> Unauthorized</title><style type="text/css">body
>>> {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b
>>> {color:white;background-co
>>> lor:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3
>>> {font-size:14px;} p {font-size:12px;} a {color:black;} .line
>>> {height:1px;background-color:#525D76;border:none;}</style></head><bod
>>> y><h1>HTTP Status 401 – Unauthorized</h1><hr class="line"
>>> /><p><b>Type</b> Status Report</p><p><b>Description</b> The request has not
>>> been applied to the target resource because it lacks va
>>> lid authentication credentials for that resource.</p><hr class="line"
>>> /><h3>Apache Tomcat</h3></body></html>
>>>
>>> So, am I right to be using the digested password? Do you see anything
>>> else that could be wrong? Why does this work for the browser but not for
>>> pydap?
>>>
>>> I will add that the algorithm for the CredentialHandler is "sha-*512*"
>>> in the ~tomcat/conf/server.xml file inside the container, so that is why
>>> the digested password is an sha512 digest. And the clear text password is
>>> "flukeTmp". I'll be changing that for our production system.
>>>
>>> And, all of this - the TDS configuration and the test python script with
>>> the above URL - are now checked in to our thredds-dpc
>>> <https://github.com/JimFluke/thredds-dpc/tree/master> repository on
>>> GitHub so you can look at the details there.
>>>
>>> Any help would be greatly appreciated.
>>>
>>> Thanks,
>>> Jim
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>> recorded in the Unidata inquiry tracking system and made publicly
>>> available through the web. Users who post to any of the lists we
>>> maintain are reminded to remove any personal information that they
>>> do not want to be made public.
>>>
>>>
>>> thredds mailing list
>>> thredds@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> https://www.unidata.ucar.edu/mailing_lists/
>>>
>>
>>
>
