fix broken doc string in rdzv module and revamp the layout of the rendezvous.html page (#87)

Summary:
Pull Request resolved: #87

see title

Differential Revision: D20896988

fbshipit-source-id: 354644348c1b4fe5a91c8d431ef23668e481a92b

Author: Kiuk Chung

docs/source/examples.rst

@@ -1,5 +1,5 @@
 Examples
-=========
+=============
 
 The examples below run on the `torchelastic/examples <https://hub.docker.com/r/torchelastic/examples>`_
 Docker image, built from the `examples/Dockerfile <https://github.com/pytorch/elastic/blob/master/examples/Dockerfile>`_.
@@ -71,12 +71,15 @@ Launch ``$NUM_CUDA_DEVICES`` number of workers on a single node:
               --batch-size 32
               /workspace/data/tiny-imagenet-200
 
-Multi-container, multi-worker
--------------------------------
+Multi-container
+----------------
 
 In this example we will launch multiple containers on a single node.
-Each container is running multiple workers.
-This demonstrates how a multi-node launch would work (each node runs a container).
+Please follow the instructions in the multi-container example
+`README <https://github.com/pytorch/elastic/tree/master/examples/multi_container/README.md>`_.
+
+Each container runs multiple workers. This demonstrates how a multi-node launch
+would work (each node runs a container occupying the whole node).
 
 The high-level differences between a single-container vs multi-container
 launches are:
@@ -85,14 +88,22 @@ launches are:
 2. An etcd server must be setup before starting the worker containers.
 3. Remove ``--with_etcd`` and specify ``--rdzv_backend``, ``--rdzv_endpoint`` and ``--rdzv_id``.
 
-For more information see `elastic launch <distributed.html>`_).
+For more information see `elastic launch <distributed.html>`_.
+
 
-<PLACEHOLDER, add multi-container example instructions here>
 
-Multi-node, multi-worker
--------------------------
+Multi-node
+-----------
 
 The multi-node, multi-worker case is similar to running multi-container, multi-worker.
 Simply run each container on a separate node, occupying the entire node.
 Alternatively, you can use our kubernetes
 `elastic job controller <kubernetes.html>`_ to launch a multi-node job.
+
+.. warning:: We recommend you setup a highly available etcd server when
+             deploying multi-node jobs in production as this is the single
+             point of failure for your jobs. Depending on your usecase
+             you can either sidecar an etcd server with each job or setup
+             a shared etcd server. If etcd does not meet your requirements
+             you can implement your own rendezvous handler and use our
+             APIs to create a custom launcher.

docs/source/rendezvous.rst

@@ -2,19 +2,54 @@ Rendezvous
 ==========
 
 .. automodule:: torchelastic.rendezvous
-.. currentmodule:: torchelastic.rendezvous
 
 Below is a state diagram describing how rendezvous works.
 
 .. image:: etcd_rdzv_diagram.png
 
+
+Handler
+--------------------
+
+.. currentmodule:: torchelastic.rendezvous
+
+.. autoclass:: RendezvousHandler
+   :members:
+
+Exceptions
+-------------
+.. autoclass:: RendezvousClosedException
+.. autoclass:: RendezvousTimeoutException
+.. autoclass:: RendezvousNonRetryableError
+
+Implmentations
+----------------
+
 Etcd Rendezvous
----------------
+****************
 
 .. currentmodule:: torchelastic.rendezvous.etcd_rendezvous
 
 .. autoclass:: EtcdRendezvousHandler
-   :members:
 
 .. autoclass:: EtcdRendezvous
    :members:
+
+.. autoclass:: EtcdStore
+   :members:
+
+Etcd Server
+*************
+
+The ``EtcdServer`` is a convenience class that makes it easy for you to
+start and stop an etcd server on a subprocess. This is useful for testing
+or single-node (multi-worker) deployments where manually setting up an
+etcd server on the side is cumbersome.
+
+.. warning:: For production and multi-node deployments please consider
+             properly deploying a highly available etcd server as this is
+             the single point of failure for your distributed jobs.
+
+.. currentmodule:: torchelastic.rendezvous.etcd_server
+
+.. autoclass:: EtcdServer

torchelastic/rendezvous/api.py

@@ -21,7 +21,8 @@ class RendezvousClosedException(Exception):
 
 class RendezvousTimeoutException(Exception):
     """
-    Raised from `next_rendezvous` to signal that the rendezvous did not
+    Raised from ``RendezvousHandler.next_rendezvous()`` to signal that the
+    rendezvous did not
     succeed within the allocated time. This is meant to be interpreted
     as a non-retryable type of failure.
     """
@@ -31,7 +32,7 @@ class RendezvousTimeoutException(Exception):
 
 class RendezvousNonRetryableError(Exception):
     """
-    Raised from any of the `RendezvousHandler` methods when a failure
+    Raised from any of the ``RendezvousHandler`` methods when a failure
     occured that should not be retried with the same worker process.
     """
 
@@ -60,12 +61,12 @@ def next_rendezvous(self) -> Tuple["torch.distributed.Store", int, int]:
         process is included in the formed worker group), or a timeout occurs, or
         rendezvous was marked closed.
 
-        Returns a tuple of (``c10d Store``, ``rank``, ``world size``)
+        Returns: a tuple of (``c10d Store``, ``rank``, ``world size``)
 
         Raises:
-            ``RendezvousClosedException`` if rendezvous for the current
+            RendezvousClosedException - if rendezvous for the current
                job is closed.
-            ``RendezvousTimeoutException`` on timeout
+            RendezvousTimeoutException - on timeout
         """
         pass
 
@@ -76,7 +77,7 @@ def is_closed(self) -> bool:
         which means all future attempts to re-rendezvous (within same job) will
         fail.
 
-        .. note:: ``is_closed``/``set_closed`` have semantics of eventual
+        .. note:: ``is_closed`` and ``set_closed`` have semantics of eventual
                   propagation, and should not be used for synchronization.
                   The intention here is that if at least one worker decides
                   the job is finished, it will close the rendezvous, and