IOError: [Errno 24] Too many open files

hvoss's Avatar

hvoss

11 May, 2017 08:02 AM

Hey,
recently i get this error more often.
The file limit for my user is set to 65535.
Still i get this error:

2017-05-11 08:49:50.273 ERROR [rhodecode.config.middleware] error occurred handling this request for path: /XXXXXX/xxxxxx/pull-request/79
Traceback (most recent call last):
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/middleware/error_handling.py", line 73, in handle_request
    response = request.get_response(self.app)
  File "/opt/rhodecode/store/cbcb7xvl643c22r0k153ggyhvzkb87z5-python2.7-WebOb-1.3.1/lib/python2.7/site-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/opt/rhodecode/store/cbcb7xvl643c22r0k153ggyhvzkb87z5-python2.7-WebOb-1.3.1/lib/python2.7/site-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/rhodecode/store/8glgrrbbbfhqc1s6nf9inbd456jjvv9w-python2.7-Paste-2.0.3/lib/python2.7/site-packages/paste/registry.py", line 379, in __call__
    app_iter = self.application(environ, start_response)
  File "/opt/rhodecode/store/hhmlr54ldv336yp0ajc4wyk97rhndmd1-python2.7-Pylons-1.0.2.rhodecode-patch1/lib/python2.7/site-packages/pylons/wsgiapp.py", line 103, in __call__
    response = self.dispatch(controller, environ, start_response)
  File "/opt/rhodecode/store/hhmlr54ldv336yp0ajc4wyk97rhndmd1-python2.7-Pylons-1.0.2.rhodecode-patch1/lib/python2.7/site-packages/pylons/wsgiapp.py", line 313, in dispatch
    return controller(environ, start_response)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/base.py", line 499, in __call__
    return WSGIController.__call__(self, environ, start_response)
  File "/opt/rhodecode/store/hhmlr54ldv336yp0ajc4wyk97rhndmd1-python2.7-Pylons-1.0.2.rhodecode-patch1/lib/python2.7/site-packages/pylons/controllers/core.py", line 214, in __call__
    response = self._dispatch_call()
  File "/opt/rhodecode/store/hhmlr54ldv336yp0ajc4wyk97rhndmd1-python2.7-Pylons-1.0.2.rhodecode-patch1/lib/python2.7/site-packages/pylons/controllers/core.py", line 164, in _dispatch_call
    response = self._inspect_call(func)
  File "/opt/rhodecode/store/hhmlr54ldv336yp0ajc4wyk97rhndmd1-python2.7-Pylons-1.0.2.rhodecode-patch1/lib/python2.7/site-packages/pylons/controllers/core.py", line 107, in _inspect_call
    result = self._perform_call(func, args)
  File "/opt/rhodecode/store/hhmlr54ldv336yp0ajc4wyk97rhndmd1-python2.7-Pylons-1.0.2.rhodecode-patch1/lib/python2.7/site-packages/pylons/controllers/core.py", line 57, in _perform_call
    return func(**args)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1969, in local_wrapper
    return wrapper(func, *args, **kwds)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1155, in __wrapper
    return func(*fargs, **fkwargs)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1969, in local_wrapper
    return wrapper(func, *args, **kwds)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1248, in __wrapper
    return func(*fargs, **fkwargs)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1969, in local_wrapper
    return wrapper(func, *args, **kwds)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1287, in __wrapper
    return func(*fargs, **fkwargs)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1969, in local_wrapper
    return wrapper(func, *args, **kwds)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1371, in __wrapper
    return func(*fargs, **fkwargs)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/controllers/pullrequests.py", line 530, in merge
    self._merge_pull_request(pull_request, user, extras)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/controllers/pullrequests.py", line 539, in _merge_pull_request
    pull_request, user, extras=extras)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/model/pull_request.py", line 508, in merge
    merge_state = self._merge_pull_request(pull_request, user, extras)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/model/pull_request.py", line 548, in _merge_pull_request
    user_email=user.email, message=message, use_rebase=use_rebase)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/base.py", line 468, in merge
    use_rebase=use_rebase)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/git/repository.py", line 863, in _merge_repo
    if target_ref.commit_id != shadow_repo.branches[target_ref.name]:
  File "/opt/rhodecode/store/a2fnbjl4zgm6gvszy1qkvgnrmwm4a1vl-python2.7-zope.cachedescriptors-4.0.0/lib/python2.7/site-packages/zope/cachedescriptors/property.py", line 69, in __get__
    value = func(inst)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/git/repository.py", line 315, in branches
    return self._get_branches()
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/git/repository.py", line 311, in _get_branches
    return self._get_refs_entries(prefix='refs/heads/', strip_prefix=True)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/git/repository.py", line 298, in _get_refs_entries
    for ref, sha in self._refs.iteritems():
  File "/opt/rhodecode/store/a2fnbjl4zgm6gvszy1qkvgnrmwm4a1vl-python2.7-zope.cachedescriptors-4.0.0/lib/python2.7/site-packages/zope/cachedescriptors/property.py", line 69, in __get__
    value = func(inst)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/git/repository.py", line 388, in _refs
    return self._get_refs()
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/git/repository.py", line 384, in _get_refs
    return self._remote.get_refs()
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/vcs/client_http.py", line 124, in f
    return self._call(name, *args, **kwargs)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/vcs/client_http.py", line 143, in _call_with_logging
    return RemoteRepo._call(self, name, *args, **kwargs)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/vcs/exceptions.py", line 179, in wrapper
    return func(*args, **kwargs)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/vcs/client_http.py", line 138, in _call
    return _remote_call(self.url, payload, EXCEPTIONS_MAP, self._session)
  File "/opt/rhodecode/store/9q264m389m52myrgjanjivaab4192bva-python2.7-rhodecode-enterprise-ce-4.7.2/lib/python2.7/site-packages/rhodecode/lib/vcs/client_http.py", line 226, in _remote_call
    raise exc
Exception
 +--- This exception occured remotely on VCSServer - Remote traceback:

Traceback (most recent call last):
  File "/opt/rhodecode/store/xpdg97yhawxyvrh90666414x459kriq3-python2.7-rhodecode-vcsserver-4.7.2/lib/python2.7/site-packages/vcsserver/http_main.py", line 243, in vcs_view
    resp = getattr(remote, method)(*args, **kwargs)
  File "/opt/rhodecode/store/xpdg97yhawxyvrh90666414x459kriq3-python2.7-rhodecode-vcsserver-4.7.2/lib/python2.7/site-packages/vcsserver/git.py", line 55, in wrapper
    return func(*args, **kwargs)
  File "/opt/rhodecode/store/xpdg97yhawxyvrh90666414x459kriq3-python2.7-rhodecode-vcsserver-4.7.2/lib/python2.7/site-packages/vcsserver/git.py", line 504, in get_refs
    peeled_sha = repo.get_peeled(ref)
  File "/opt/rhodecode/store/njvh3kvqba0ivdji5z7h1gs0qay04kcx-python2.7-dulwich-0.13.0/lib/python2.7/site-packages/dulwich/repo.py", line 403, in get_peeled
  File "/opt/rhodecode/store/njvh3kvqba0ivdji5z7h1gs0qay04kcx-python2.7-dulwich-0.13.0/lib/python2.7/site-packages/dulwich/refs.py", line 227, in __getitem__
  File "/opt/rhodecode/store/njvh3kvqba0ivdji5z7h1gs0qay04kcx-python2.7-dulwich-0.13.0/lib/python2.7/site-packages/dulwich/refs.py", line 209, in _follow
  File "/opt/rhodecode/store/njvh3kvqba0ivdji5z7h1gs0qay04kcx-python2.7-dulwich-0.13.0/lib/python2.7/site-packages/dulwich/refs.py", line 185, in read_ref
  File "/opt/rhodecode/store/njvh3kvqba0ivdji5z7h1gs0qay04kcx-python2.7-dulwich-0.13.0/lib/python2.7/site-packages/dulwich/refs.py", line 503, in read_loose_ref
  File "/opt/rhodecode/store/njvh3kvqba0ivdji5z7h1gs0qay04kcx-python2.7-dulwich-0.13.0/lib/python2.7/site-packages/dulwich/file.py", line 87, in GitFile
IOError: [Errno 24] Too many open files: '/home/rhodecode/repos/XXXXX/.__shadow_XXXXX_pr-79/.git/refs/heads/pr_feature/10684-develop_1'

 +--- End of remote traceback

In my frontend it just shows a HTTP 500 Error page.

  1. Support Staff 1 Posted by Marcin Kuzminsk... on 11 May, 2017 08:40 AM

    Marcin Kuzminski's Avatar

    Hi,

    It could be related to big repository fragmentation, please go to /home/rhodecode/repos/XXXXX/.__shadow_XXXXX_pr-79/.git/refs/heads/pr_feature/10684-develop_1 and run git gc

    Also on the TARGET repo, there's a repository maintainance under settings, which should be also executed.

    Hope that helps

  2. 2 Posted by hvoss on 11 May, 2017 08:43 AM

    hvoss's Avatar

    I did it, this happens on more than one repository.
    I'll run the maintaince in every repo and hope it will not continue

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac