SSL problem when updating using rcccontrol

SSL problem with rccontrol's Avatar

SSL problem with rccontrol

28 Sep, 2016 02:57 AM

When I try to update rccontrol, I get the following error message:

$ rccontrol self-update
Currently running "1.6.5"
Downloading https://dls.rhodecode.com/linux/MANIFEST ...

Failed downloading (https://dls.rhodecode.com/linux/MANIFEST) due to "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)"

Network related problem encountered.

If you are in an offline environment please run with `--offline`.

Further details available at https://docs.rhodecode.com/RhodeCode-Control/tasks/upgrade-rcc.html

--
I can update using offline methods, but how can I fix this problem?

  1. Support Staff 1 Posted by Marcin Kuzminsk... on 28 Sep, 2016 08:24 AM

    Marcin Kuzminski's Avatar

    Hi,

    This is very odd. We did upgrade and restart our download servers to address latest OpenSSL problems (https://www.openssl.org/news/secadv/20160926.txt)

    We tried the upgrade/self-update from few of our servers and we don't see any SSL issues. Does Sandia use any special certificates or require any special SSL version/protocol ?

    Cheers

  2. 2 Posted by Haagen, Christo... on 28 Sep, 2016 04:48 PM

    Haagen, Christopher M's Avatar

    There is an http proxy and a reverse SSL (interception) proxy, but I can, for example, download the MANIFEST file using a web browser. Maybe I need to add the SSL interception proxy certificate to the CA certs. I know where to get the certificate, but I’m not sure how to add it to the server or RC configuration.

    -Chris

    Chris Haagen
    Sandia National Laboratories
    phone (925) 294-3359
    email [email blocked]<mailto:[email blocked]>

    On Sep 28, 2016, at 01:24, Marcin Kuzminski <[email blocked]<mailto:[email blocked]>> wrote:

  3. Support Staff 3 Posted by Marcin Kuzminsk... on 28 Sep, 2016 04:57 PM

    Marcin Kuzminski's Avatar

    Chris-

    Please take a look at ~/.rccontrol/supervisor/supervisord.ini we already add there an SSL certificate via env variable, probably you could change it there.

    Cheers

  4. 4 Posted by Haagen, Christo... on 28 Sep, 2016 06:32 PM

    Haagen, Christopher M's Avatar

    I downloaded the SSL interception certificate (in .pem format) and appended (cat $file.pem >> /home/$user/.rccontrol-profile/etc/ca-bundle.crt) it to the crt file:

    /home/$user/.rccontrol-profile/etc/ca-bundle.crt

    I still get the same error when I run rccontrol self-update.

    Are there any log files I can look at for more information about this issue?

    -Chris

    Chris Haagen
    Sandia National Laboratories
    phone (925) 294-3359
    email [email blocked] <mailto:[email blocked]>

  5. Support Staff 5 Posted by Marcin Kuzminsk... on 28 Sep, 2016 06:42 PM

    Marcin Kuzminski's Avatar

    Hmm, ok then it might be that this doesn't get applied, have you tried to exporting the same evn variable before running rccontrol update ?

    kind of like SSL_CERT=/path rccontrol self-update

  6. 6 Posted by Haagen, Christo... on 28 Sep, 2016 07:43 PM

    Haagen, Christopher M's Avatar

    I tried this:

    $ SSL_CERT_FILE=/home/rcuser/.rccontrol-profile/etc/ca-bundle.crt rccontrol self-update
    Currently running "1.6.5"
    Downloading https://dls.rhodecode.com/linux/MANIFEST ...

    Failed downloading (https://dls.rhodecode.com/linux/MANIFEST) due to "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)"

    Network related problem encountered.

    If you are in an offline environment please run with `--offline`.

    Further details available at https://docs.rhodecode.com/RhodeCode-Control/tasks/upgrade-rcc.html <https://docs.rhodecode.com/RhodeCode-Control/tasks/upgrade-rcc.html>

    -Chris

    Chris Haagen
    Sandia National Laboratories
    phone (925) 294-3359
    email [email blocked] <mailto:[email blocked]>

  7. Support Staff 7 Posted by Marcin Kuzminsk... on 29 Sep, 2016 10:19 AM

    Marcin Kuzminski's Avatar

    Hmm, i think we need to investigate this.

    Are there any changes made in your system recently ?

    I know for sure we updated python version recently, and that have now much stricter SSL checks, maybe that is related.

  8. 8 Posted by Haagen, Christo... on 29 Sep, 2016 05:23 PM

    Haagen, Christopher M's Avatar

    No recent changes. I have no problem downloading the MANIFEST or other files from the same URL using cURL on the same machine.

    I’m happy to look at any logs or try any tests you recommend.

    -Chris

    Chris Haagen
    Sandia National Laboratories
    phone (925) 294-3359
    email [email blocked] <mailto:[email blocked]>

  9. 9 Posted by Haagen, Christo... on 29 Sep, 2016 05:36 PM

    Haagen, Christopher M's Avatar

    I only recently installed RhodeCode (starting with 4.3.1), so the self-update never worked for me (for example, on a previous version). Yesterday I manually updated everything to RCC 1.6.5 and Community/VCS 4.4.1.

    Another thing that does not work is “check for updates” link on the Admin > Settings > System Info page (/_admin/settings/system) of the web GUI. I don’t know if that is related.

    I’m sure this is unrelated, but on Safari v10.0, the System Info page does not display any content — only a blank container. Works fine on Chrome, Firefox, etc.

    -Chris

    Chris Haagen
    Sandia National Laboratories
    phone (925) 294-3359
    email [email blocked] <mailto:[email blocked]>

  10. Support Staff 10 Posted by Marcin Kuzminsk... on 29 Sep, 2016 06:36 PM

    Marcin Kuzminski's Avatar

    I looked at the code, and we use this library to download the MANIFEST file, it uses requests.

    Here are the docs: http://docs.python-requests.org/en/master/user/advanced/#ssl-cert-v...

    I found that we might try to set:
    CA_BUNDLE or REQUESTS_CA_BUNDLE

    as of this part: This list of trusted CAs can also be specified through the REQUESTS_CA_BUNDLE environment variable.

    Could you try to set those two env variables on the self-update call ?

  11. 11 Posted by Haagen, Christo... on 29 Sep, 2016 06:43 PM

    Haagen, Christopher M's Avatar

    It works when REQUESTS_CA_BUNDLE is set:

    [rcuser@as03callx ~]$ CA_BUNDLE=/home/rcuser/.rccontrol-profile/etc/ca-bundle.crt rccontrol self-update
    Currently running "1.6.5"
    Downloading https://dls.rhodecode.com/linux/MANIFEST ...

    Failed downloading (https://dls.rhodecode.com/linux/MANIFEST) due to "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)"

    Network related problem encountered.

    If you are in an offline environment please run with `--offline`.

    Further details available at https://docs.rhodecode.com/RhodeCode-Control/tasks/upgrade-rcc.html
    [rcuser@as03callx ~]$ REQUESTS_CA_BUNDLE=/home/rcuser/.rccontrol-profile/etc/ca-bundle.crt rccontrol self-update
    Currently running "1.6.5"
    Downloading https://dls.rhodecode.com/linux/MANIFEST ...

    Latest version "1.6.5"
    No new version of RhodeCode Control available.

    Is there a way to fix this problem more permanently, and would the same issue affect the “check for updates” link?

    -Chris

    Chris Haagen
    Sandia National Laboratories
    phone (925) 294-3359
    email [email blocked] <mailto:[email blocked]>

  12. Support Staff 12 Posted by Marcin Kuzminsk... on 29 Sep, 2016 09:03 PM

    Marcin Kuzminski's Avatar

    For the web iterface please add this env variable into supervisord as i already mentioned and restart supervisord process.

    AS permanent fix i'd simply create an alias for rccontrol

    alias rccontrol="REQUESTS_CA_BUNDLE=/home/rcuser/.rccontrol-profile/etc/ca-bundle.crt rccontrol"

    Hope that helps

  13. 13 Posted by Haagen, Christo... on 29 Sep, 2016 09:27 PM

    Haagen, Christopher M's Avatar

    Thanks. For the supervisor ini file, there are many sections that have “environment=“ keys. which one(s) should be updated?

    -Chris

    Chris Haagen
    Sandia National Laboratories
    phone (925) 294-3359
    email [email blocked] <mailto:[email blocked]>

  14. 14 Posted by Haagen, Christo... on 30 Sep, 2016 01:58 AM

    Haagen, Christopher M's Avatar

    OK, everything seems to work OK with the web interface, EXCEPT when started from the systemd unit:

    [Unit]
    Description=Rhodecode

    [Service]
    Type=forking
    User=rcuser
    Environment="REQUESTS_CA_BUNDLE=/home/rcuser/.rccontrol-profile/etc/ca-bundle.crt"
    ExecStart=/home/rcuser/.rccontrol-profile/bin/rccontrol-self-init
    PIDFile=/home/rcuser/.rccontrol/supervisor/supervisord.pid

    [Install]
    WantedBy=multi-user.target

    Any ideas?

    -Chris

    Chris Haagen
    Sandia National Laboratories
    phone (925) 294-3359
    email [email blocked] <mailto:[email blocked]>

  15. Support Staff 15 Posted by Marcin Kuzminsk... on 30 Sep, 2016 09:42 AM

    Marcin Kuzminski's Avatar

    Hi Chris,

    You should update all entries in supervisord, we'll add this fix into the next control release so it adds this automatically for users.

    I'm not sure if i fully understand the systemd problem. Is there any error ?

  16. Support Staff 16 Posted by Marcin Kuzminsk... on 13 Oct, 2016 04:27 PM

    Marcin Kuzminski's Avatar

    We released new control that should fix this particular issue permenently. It now automatically adds the REQUESTS_CA_BUNDLE into new instances.

  17. Marcin Kuzminski closed this discussion on 13 Oct, 2016 04:27 PM.

  18. Haagen, Christopher M re-opened this discussion on 13 Oct, 2016 06:27 PM

  19. 17 Posted by Haagen, Christo... on 13 Oct, 2016 06:27 PM

    Haagen, Christopher M's Avatar

    I updated rccontrol, but this does not seem to fix the web “check for updates” link.

    However, this does make the web link work:

    [rcuser@as03callx ~]$ REQUESTS_CA_BUNDLE=/home/rcuser/.rccontrol-profile/etc/ca-bundle.crt rccontrol-self-init

    after I appended my SSL interception certificate to the bundle.

    -Chris

    Chris Haagen
    Sandia National Laboratories
    phone (925) 294-3359
    email [email blocked] <mailto:[email blocked]>

  20. Support Staff 18 Posted by Marcin Kuzminsk... on 13 Oct, 2016 06:31 PM

    Marcin Kuzminski's Avatar

    The exact env variable should be now added to the enterprise entries. Unless you have enabled self-managed supervisor flag, then it's not added.

    Here's how an new entry looks like with new control:

    [program:enterprise-4_script]
    numprocs = 1
    redirect_stderr = true
    _port = 10007
    environment = PYTHONPATH="",SSL_CERT_FILE="/home/ubuntu/.rccontrol-profile/etc/ca-bundle.crt",REQUESTS_CA_BUNDLE="/home/ubuntu/.rccontrol-profile/etc/ca-bundle.crt",GIT_SSL_CAINFO="/home/ubuntu/.rccontrol-profile/etc/ca-bundle.crt"
    _host = 0.0.0.0
    command = /home/ubuntu/.rccontrol/enterprise-4/profile/bin/gunicorn --error-logfile=- --paster=/home/ubuntu/.rccontrol/enterprise-4/rhodecode.ini
    autostart = true
    directory = /home/ubuntu/.rccontrol/enterprise-4
    stdout_logfile = /home/ubuntu/.rccontrol/enterprise-4/enterprise.log
    

    It's odd that you need to add this... can you check if this is how your enviroment= looks like in supervisord.ini ?

    Cheers

  21. 19 Posted by Haagen, Christo... on 13 Oct, 2016 09:35 PM

    Haagen, Christopher M's Avatar

    Here is my supervisord.ini file:

    [rcuser ~]$ cat .rccontrol/supervisor/supervisord.ini
    [supervisord]
    minfds = 1024
    minprocs = 200
    loglevel = info
    environment = HOME=/home/rcuser,LANG=en_US.UTF-8
    strip_ansi = true
    logfile = /home/rcuser/.rccontrol/supervisor/supervisord.log
    pidfile = /home/rcuser/.rccontrol/supervisor/supervisord.pid
    [supervisorctl]
    username = rccontrol-admin
    password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    prompt = RhodeCode Control Supervisor
    serverurl = http://127.0.0.1:10000
    [inet_http_server]
    username = rccontrol-admin
    password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    port = 127.0.0.1:10000
    _port = 10000
    [include]
    files = rhodecode_config*.ini
    [rpcinterface:supervisor]
    supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
    [group:community-1]
    programs = community-1_script
    [program:community-1_script]
    numprocs = 1
    redirect_stderr = true
    _port = 10002
    environment = PYTHONPATH="",SSL_CERT_FILE="/home/rcuser/.rccontrol-profile/etc/ca-bundle.crt",REQUESTS_CA_BUNDLE="/home/rcuser/.rccontrol-profile/etc/ca-bundle.crt",GIT_SSL_CAINFO="/home/rcuser/.rccontrol-profile/etc/ca-bundle.crt"
    _host = 127.0.0.1
    command = /home/rcuser/.rccontrol/community-1/profile/bin/gunicorn --error-logfile=- --paster=/home/rcuser/.rccontrol/community-1/rhodecode.ini
    autostart = true
    directory = /home/rcuser/.rccontrol/community-1
    stdout_logfile = /home/rcuser/.rccontrol/community-1/community.log
    [group:vcsserver-1]
    programs = vcsserver-1_script
    [program:vcsserver-1_script]
    numprocs = 1
    redirect_stderr = true
    _port = 10001
    environment = SSL_CERT_FILE="/home/rcuser/.rccontrol-profile/etc/ca-bundle.crt",REQUESTS_CA_BUNDLE="/home/rcuser/.rccontrol-profile/etc/ca-bundle.crt"
    _host = 127.0.0.1
    command = /home/rcuser/.rccontrol/vcsserver-1/profile/bin/vcsserver --config=/home/rcuser/.rccontrol/vcsserver-1/vcsserver.ini
    autostart = true
    directory = /home/rcuser/.rccontrol/vcsserver-1
    stdout_logfile = /home/rcuser/.rccontrol/vcsserver-1/vcsserver.log

    -Chris

    Chris Haagen
    Sandia National Laboratories
    phone (925) 294-3359
    email [email blocked] <mailto:[email blocked]>

  22. Support Staff 20 Posted by Marcin Kuzminsk... on 14 Oct, 2016 09:00 AM

    Marcin Kuzminski's Avatar

    can you check your ~/.rccontrol.ini file ?

    If there is nothing about self_managed_supervisord=True This means some of the flags should change when you completely restart supervisord. Did you try that ?

  23. 21 Posted by Haagen, Christo... on 17 Oct, 2016 09:56 PM

    Haagen, Christopher M's Avatar

    [rcuser@as03callx ~]$ cat .rccontrol.ini
    [instance:vcsserver-1]
    start_at_boot = True
    self_managed_supervisor = False
    [instance:community-1]
    start_at_boot = True
    self_managed_supervisor = False

    Should this be something different?

    -Chris

    Chris Haagen
    Sandia National Laboratories
    phone (925) 294-3359
    email [email blocked] <mailto:[email blocked]>

  24. Support Staff 22 Posted by Marcin Kuzminsk... on 17 Oct, 2016 10:03 PM

    Marcin Kuzminski's Avatar

    This looks ok.

    I think you need to restart the whole supervisord in order for changes to be re-applied.
    You can do so via kill $(cat ~/.rccontrol/supervisord/supervisor.pid)

    And then rccontrol self-init. I think that will update the ENV configuration

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac