waveform indexer: stops working when SQL/SSL connection gets closed unexpectedly

[megies]

When something unexpected happens to the database connection the indexer processes that are running in background forever stop indexing new waveforms but the processes keep running. Ideally database connection should be reestablished, or admin should get notified of the issue.

Traceback:

Traceback (most recent call last):
  File "/local_home/jane/jane/src/jane/waveforms/management/commands/index_waveforms.py", line 229, in iterate
    file = self._current_files.pop(0)
IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/local_home/jane/.local/lib/python3.4/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
psycopg2.OperationalError: terminating connection due to administrator command
SSL connection has been closed unexpectedly


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/local_home/jane/jane/src/manage.py", line 12, in <module>
    execute_from_command_line(sys.argv)
  File "/local_home/jane/.local/lib/python3.4/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "/local_home/jane/.local/lib/python3.4/site-packages/django/core/management/__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/local_home/jane/.local/lib/python3.4/site-packages/django/core/management/base.py", line 348, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/local_home/jane/.local/lib/python3.4/site-packages/django/core/management/base.py", line 399, in execute
    output = self.handle(*args, **options)
  File "/local_home/jane/jane/src/jane/waveforms/management/commands/index_waveforms.py", line 492, in handle
    _run_indexer(kwargs)
  File "/local_home/jane/jane/src/jane/waveforms/management/commands/index_waveforms.py", line 405, in _run_indexer
    service.serve_forever(options["poll_interval"])
  File "/local_home/jane/jane/src/jane/waveforms/management/commands/index_waveforms.py", line 364, in serve_forever
    self.iterate()
  File "/local_home/jane/jane/src/jane/waveforms/management/commands/index_waveforms.py", line 237, in iterate
    self._step_walker()
  File "/local_home/jane/jane/src/jane/waveforms/management/commands/index_waveforms.py", line 183, in _step_walker
    self._db_files = self._select(self._current_path)
  File "/local_home/jane/jane/src/jane/waveforms/management/commands/index_waveforms.py", line 68, in _select
    return {i[0]: i[1] for i in files}
  File "/local_home/jane/.local/lib/python3.4/site-packages/django/db/models/query.py", line 258, in __iter__
    self._fetch_all()
  File "/local_home/jane/.local/lib/python3.4/site-packages/django/db/models/query.py", line 1074, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/local_home/jane/.local/lib/python3.4/site-packages/django/db/models/query.py", line 128, in __iter__
    for row in compiler.results_iter():
  File "/local_home/jane/.local/lib/python3.4/site-packages/django/db/models/sql/compiler.py", line 802, in results_iter
    results = self.execute_sql(MULTI)
  File "/local_home/jane/.local/lib/python3.4/site-packages/django/db/models/sql/compiler.py", line 848, in execute_sql
    cursor.execute(sql, params)
  File "/local_home/jane/.local/lib/python3.4/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/local_home/jane/.local/lib/python3.4/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/local_home/jane/.local/lib/python3.4/site-packages/django/db/utils.py", line 95, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/local_home/jane/.local/lib/python3.4/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/local_home/jane/.local/lib/python3.4/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
django.db.utils.OperationalError: terminating connection due to administrator command
SSL connection has been closed unexpectedly

CC @jwassermann