Fix installation issues under python 3.7

AUTHORS

@@ -1 +1,2 @@
 Vikash Singh <vikash.duliajan@gmail.com>
+amrrs <amrrs@outlook.com>

ChangeLog

@@ -1,6 +1,22 @@
 CHANGES
 =======
 
+* fix for https://github.com/vi3k6i5/GuidedLDA/issues/5 Random State seed was not working for document topic initialisation for seeded topics
+* fixed typos
+* fixed wrong md formatting
+* fixing typos
+* add link to dataset nyt
+* added spacing in readme
+* added package version and doc build status
+* corrected documentation
+* corrected doc file path
+* corrected path for dataset
+* fixed docs path
+* corrected docs
+* updated guided lda seeded lda description
+* fixed code sample
+* temporary fix for installation process
+* changed installer to reflect lda library installer
 * in progress
 * updated package name
 * initial commit

README.rst

@@ -1,30 +1,50 @@
 GuidedLDA: Guided Topic modeling with latent Dirichlet allocation
 ====================================================
 
-``guidedlda`` implements latent Dirichlet allocation (LDA) using collapsed Gibbs
-sampling. ``guidedlda`` is fast and is tested on Linux, OS X, and Windows.
+.. image:: https://readthedocs.org/projects/guidedlda/badge/?version=latest
+    :target: http://guidedlda.readthedocs.io/en/latest/?badge=latest
+    :alt: Documentation Status
 
-You can read more about lda in `the documentation <https://guidedlda.readthedocs.io>`_.
+.. image:: https://badge.fury.io/py/guidedlda.svg
+    :target: https://badge.fury.io/py/guidedlda
+    :alt: Package version
+
+
+``GuidedLDA`` OR ``SeededLDA`` implements latent Dirichlet allocation (LDA) using collapsed Gibbs sampling. ``GuidedLDA`` can be guided by setting some seed words per topic. Which will make the topics converge in that direction.
+
+You can read more about guidedlda in `the documentation <https://guidedlda.readthedocs.io>`_.
+
+I published an article about it on `freecodecamp Medium blog <https://medium.freecodecamp.org/how-we-changed-unsupervised-lda-to-semi-supervised-guidedlda-e36a95f3a164>`_.
 
 Installation
 ------------
 
-.. code-block:: bash
+::
+
+    pip install guidedlda
+
+If pip install does not work, then try the next step:
+
+::
+
     https://github.com/vi3k6i5/GuidedLDA
     cd GuidedLDA
+    sh build_dist.sh
+    python setup.py sdist
     pip install -e .
 
-
-
-Pip install is not working right now are is being fixed. Once that is fixed you should be able to simply do ``pip install guidedlda``.
+If the above step also does not work, please raise an `issue <https://github.com/vi3k6i5/guidedlda/issues>`_ with details of your workstation's OS version, Python version, architecture etc. and I will try my best to fix it ASAP.
 
 Getting started
 ---------------
 
-``guidedlda.LDA`` implements latent Dirichlet allocation (LDA). The interface follows
+``guidedlda.GuidedLDA`` implements latent Dirichlet allocation (LDA). The interface follows
 conventions found in scikit-learn_.
 
-The following demonstrates how to inspect a model of a subset of the Reuters
+`Example Code <https://github.com/vi3k6i5/GuidedLDA/blob/master/examples/example_seeded_lda.py>`_.
+
+
+The following demonstrates how to inspect a model of a subset of the NYT
 news dataset. The input below, ``X``, is a document-term matrix (sparse matrices
 are accepted).
 
@@ -72,8 +92,8 @@ are accepted).
     >>> # Guided LDA with seed topics.
     >>> seed_topic_list = [['game', 'team', 'win', 'player', 'season', 'second', 'victory'],
     >>>                    ['percent', 'company', 'market', 'price', 'sell', 'business', 'stock', 'share'],
-    >>>                   ['music', 'write', 'art', 'book', 'world', 'film'],
-    >>>                   ['political', 'government', 'leader', 'official', 'state', 'country', 'american','case', 'law', 'police', 'charge', 'officer', 'kill', 'arrest', 'lawyer']]
+    >>>                    ['music', 'write', 'art', 'book', 'world', 'film'],
+    >>>                    ['political', 'government', 'leader', 'official', 'state', 'country', 'american','case', 'law', 'police', 'charge', 'officer', 'kill', 'arrest', 'lawyer']]
 
     >>> model = guidedlda.GuidedLDA(n_topics=5, n_iter=100, random_state=7, refresh=20)
 
@@ -126,15 +146,21 @@ The document-topic distributions should be retrived as: ``doc_topic = model.tran
     top topic: 1 Document: company, comic, series, case, executive
     top topic: 3 Document: son, scene, charge, episode, attack
 
-Save the model for production or for running later:
+Optionally, reduce the model by purging additional matrices:
 
 .. code-block:: python
-    >>> from six.moves import cPickle as pickle
-    >>> # Uncomment next step if you want to lighten the model object
+
+    >>> # Next step will lighten the model object
     >>> # This step will delete some matrices inside the model.
     >>> # you will be able to use model.transform(X) the same way as earlier.
     >>> # you wont be able to use model.fit_transform(X_new)
-    >>> # model.purge_extra_matrices()
+    >>> model.purge_extra_matrices()
+
+Save the model for production or for running later:
+
+.. code-block:: python
+
+    >>> from six.moves import cPickle as pickle
     >>> with open('guidedlda_model.pickle', 'wb') as file_handle:
     >>>     pickle.dump(model, file_handle)
     >>> # load the model for prediction
@@ -154,7 +180,7 @@ Python 2.7 or Python 3.3+ is required. The following packages are required
 Caveat
 ------
 
-``guidedlda`` aims for Guiding LDA. More often then not the topics we get from a LDA model are not to our setisfaction. GuidedLDA can give the topics a nudge in the direction we want it to converge. We have production trained it for half a million documents (We have a big machine). We have run predictions on millions and manually checked topics for thousands (we are sattisfied with the results).
+``guidedlda`` aims for Guiding LDA. More often then not the topics we get from a LDA model are not to our satisfaction. GuidedLDA can give the topics a nudge in the direction we want it to converge. We have production trained it for half a million documents (We have a big machine). We have run predictions on millions and manually checked topics for thousands (we are satisfied with the results).
 
 If you are working with a very large corpus you may wish to use more sophisticated topic models such as those implemented in hca_ and MALLET_.  hca_ is written entirely in C and MALLET_ is written in Java. Unlike ``guidedlda``, hca_ can use more than one processor at a time. Both MALLET_ and hca_ implement topic models known to be more robust than standard latent Dirichlet allocation.
 
@@ -180,14 +206,14 @@ Other implementations
 
 Credits
 -------
-I would like to thank Creators of LDA project: https://github.com/lda-project/lda.
-I used the code from that LDA project as base to implement GuidedLDA on top of it.
-Special thanks to : https://twitter.com/ariddell, https://twitter.com/tdhopper :)
+I would like to thank the creators of `LDA project <https://github.com/lda-project/lda>`_. I used the code from that LDA project as base to implement GuidedLDA on top of it.
+
+Thanks to : `Allen Riddell <https://twitter.com/ariddell>`_ and `Tim Hopper <https://twitter.com/tdhopper>`_. :)
 
 License
 -------
 
-guidedlda is licensed under Version 2.0 of the Mozilla Public License.
+``guidedlda`` is licensed under Version 2.0 of the Mozilla Public License.
 
 .. _Python: http://www.python.org/
 .. _scikit-learn: http://scikit-learn.org
@@ -199,4 +225,4 @@ guidedlda is licensed under Version 2.0 of the Mozilla Public License.
 .. _Blei et al. (2003): http://jmlr.org/papers/v3/blei03a.html
 .. _Pritchard et al. (2000): http://www.genetics.org/content/155/2/945.full
 .. _Griffiths and Steyvers (2004): http://www.pnas.org/content/101/suppl_1/5228.abstract
-.. _Jagadeesh Jagarlamudi Hal Daume III and Raghavendra Udupa (2012): http://www.umiacs.umd.edu/~jags/pdfs/GuidedLDA.pdf
+.. _Jagadeesh Jagarlamudi, Hal Daume III and Raghavendra Udupa (2012): http://www.aclweb.org/anthology/E12-1021

docs/index.rst

@@ -10,30 +10,50 @@ Welcome to GuidedLDA's documentation!
    :maxdepth: 2
    :caption: Contents:
 
-``guidedlda`` implements latent Dirichlet allocation (LDA) using collapsed Gibbs
-sampling. ``guidedlda`` is fast and is tested on Linux, OS X, and Windows.
+.. image:: https://readthedocs.org/projects/guidedlda/badge/?version=latest
+    :target: http://guidedlda.readthedocs.io/en/latest/?badge=latest
+    :alt: Documentation Status
 
-You can read more about lda in `the documentation <https://guidedlda.readthedocs.io>`_.
+.. image:: https://badge.fury.io/py/guidedlda.svg
+    :target: https://badge.fury.io/py/guidedlda
+    :alt: Package version
+
+
+``GuidedLDA`` OR ``SeededLDA`` implements latent Dirichlet allocation (LDA) using collapsed Gibbs sampling. ``GuidedLDA`` can be guided by setting some seed words per topic. Which will make the topics converge in that direction.
+
+You can read more about guidedlda in `the documentation <https://guidedlda.readthedocs.io>`_.
+
+I published an article about it on `freecodecamp Medium blog <https://medium.freecodecamp.org/how-we-changed-unsupervised-lda-to-semi-supervised-guidedlda-e36a95f3a164>`_.
 
 Installation
 ------------
 
-.. code-block:: bash
+::
+
+    pip install guidedlda
+
+If pip install does not work, then try the next step:
+
+::
+
     https://github.com/vi3k6i5/GuidedLDA
     cd GuidedLDA
+    sh build_dist.sh
+    python setup.py sdist
     pip install -e .
 
-
-
-Pip install is not working right now are is being fixed. Once that is fixed you should be able to simply do ``pip install guidedlda``.
+If the above step also does not work, please raise an `issue <https://github.com/vi3k6i5/guidedlda/issues>`_ with details of your workstation's OS version, Python version, architecture etc. and I will try my best to fix it ASAP.
 
 Getting started
 ---------------
 
-``guidedlda.LDA`` implements latent Dirichlet allocation (LDA). The interface follows
+``guidedlda.GuidedLDA`` implements latent Dirichlet allocation (LDA). The interface follows
 conventions found in scikit-learn_.
 
-The following demonstrates how to inspect a model of a subset of the Reuters
+`Example Code <https://github.com/vi3k6i5/GuidedLDA/blob/master/examples/example_seeded_lda.py>`_.
+
+
+The following demonstrates how to inspect a model of a subset of the NYT
 news dataset. The input below, ``X``, is a document-term matrix (sparse matrices
 are accepted).
 
@@ -51,11 +71,9 @@ are accepted).
 
     >>> X.sum()
     1221626
-
+    >>> # Normal LDA without seeding
     >>> model = guidedlda.GuidedLDA(n_topics=5, n_iter=100, random_state=7, refresh=20)
     >>> model.fit(X)
-    >>> topic_word = model.topic_word_
-    >>> n_top_words = 8
     INFO:guidedlda:n_documents: 8447
     INFO:guidedlda:vocab_size: 3012
     INFO:guidedlda:n_words: 1221626
@@ -69,6 +87,8 @@ are accepted).
     INFO:guidedlda:<80> log likelihood: -9617962
     INFO:guidedlda:<99> log likelihood: -9604031
 
+    >>> topic_word = model.topic_word_
+    >>> n_top_words = 8
     >>> for i, topic_dist in enumerate(topic_word):
     >>>     topic_words = np.array(vocab)[np.argsort(topic_dist)][:-(n_top_words+1):-1]
     >>>     print('Topic {}: {}'.format(i, ' '.join(topic_words)))
@@ -78,11 +98,11 @@ are accepted).
     Topic 3: place open small house music turn large play
     Topic 4: official state government political states issue leader case
 
-    >>> seed_topic_list
+    >>> # Guided LDA with seed topics.
     >>> seed_topic_list = [['game', 'team', 'win', 'player', 'season', 'second', 'victory'],
     >>>                    ['percent', 'company', 'market', 'price', 'sell', 'business', 'stock', 'share'],
-    >>>                   ['music', 'write', 'art', 'book', 'world', 'film'],
-    >>>                   ['political', 'government', 'leader', 'official', 'state', 'country', 'american','case', 'law', 'police', 'charge', 'officer', 'kill', 'arrest', 'lawyer']]
+    >>>                    ['music', 'write', 'art', 'book', 'world', 'film'],
+    >>>                    ['political', 'government', 'leader', 'official', 'state', 'country', 'american','case', 'law', 'police', 'charge', 'officer', 'kill', 'arrest', 'lawyer']]
 
     >>> model = guidedlda.GuidedLDA(n_topics=5, n_iter=100, random_state=7, refresh=20)
 
@@ -135,15 +155,21 @@ The document-topic distributions should be retrived as: ``doc_topic = model.tran
     top topic: 1 Document: company, comic, series, case, executive
     top topic: 3 Document: son, scene, charge, episode, attack
 
-Save the model for production or for running later:
+Optionally, reduce the model by purging additional matrices:
 
 .. code-block:: python
-    >>> from six.moves import cPickle as pickle
-    >>> # Uncomment next step if you want to lighten the model object
+
+    >>> # Next step will lighten the model object
     >>> # This step will delete some matrices inside the model.
     >>> # you will be able to use model.transform(X) the same way as earlier.
     >>> # you wont be able to use model.fit_transform(X_new)
-    >>> # model.purge_extra_matrices()
+    >>> model.purge_extra_matrices()
+
+Save the model for production or for running later:
+
+.. code-block:: python
+
+    >>> from six.moves import cPickle as pickle
     >>> with open('guidedlda_model.pickle', 'wb') as file_handle:
     >>>     pickle.dump(model, file_handle)
     >>> # load the model for prediction
@@ -163,7 +189,7 @@ Python 2.7 or Python 3.3+ is required. The following packages are required
 Caveat
 ------
 
-``guidedlda`` aims for Guiding LDA. More often then not the topics we get from a LDA model are not to our setisfaction. GuidedLDA can give the topics a nudge in the direction we want it to converge. We have production trained it for half a million documents (We have a big machine). We have run predictions on millions and manually checked topics for thousands (we are sattisfied with the results).
+``guidedlda`` aims for Guiding LDA. More often then not the topics we get from a LDA model are not to our satisfaction. GuidedLDA can give the topics a nudge in the direction we want it to converge. We have production trained it for half a million documents (We have a big machine). We have run predictions on millions and manually checked topics for thousands (we are satisfied with the results).
 
 If you are working with a very large corpus you may wish to use more sophisticated topic models such as those implemented in hca_ and MALLET_.  hca_ is written entirely in C and MALLET_ is written in Java. Unlike ``guidedlda``, hca_ can use more than one processor at a time. Both MALLET_ and hca_ implement topic models known to be more robust than standard latent Dirichlet allocation.
 
@@ -189,14 +215,14 @@ Other implementations
 
 Credits
 -------
-I would like to thank Creators of LDA project: https://github.com/lda-project/lda.
-I used the code from that LDA project as base to implement GuidedLDA on top of it.
-Special thanks to : https://twitter.com/ariddell, https://twitter.com/tdhopper :)
+I would like to thank the creators of `LDA project <https://github.com/lda-project/lda>`_. I used the code from that LDA project as base to implement GuidedLDA on top of it.
+
+Thanks to : `Allen Riddell <https://twitter.com/ariddell>`_ and `Tim Hopper <https://twitter.com/tdhopper>`_. :)
 
 License
 -------
 
-guidedlda is licensed under Version 2.0 of the Mozilla Public License.
+``guidedlda`` is licensed under Version 2.0 of the Mozilla Public License.
 
 .. _Python: http://www.python.org/
 .. _scikit-learn: http://scikit-learn.org
@@ -208,7 +234,7 @@ guidedlda is licensed under Version 2.0 of the Mozilla Public License.
 .. _Blei et al. (2003): http://jmlr.org/papers/v3/blei03a.html
 .. _Pritchard et al. (2000): http://www.genetics.org/content/155/2/945.full
 .. _Griffiths and Steyvers (2004): http://www.pnas.org/content/101/suppl_1/5228.abstract
-.. _Jagadeesh Jagarlamudi Hal Daume III and Raghavendra Udupa (2012): http://www.umiacs.umd.edu/~jags/pdfs/GuidedLDA.pdf
+.. _Jagadeesh Jagarlamudi, Hal Daume III and Raghavendra Udupa (2012): http://www.umiacs.umd.edu/~jags/pdfs/GuidedLDA.pdf
 
 
 Indices and tables

examples/example_seeded_lda.py

@@ -0,0 +1,41 @@
+import numpy as np
+import guidedlda
+
+X = guidedlda.datasets.load_data(guidedlda.datasets.NYT)
+vocab = guidedlda.datasets.load_vocab(guidedlda.datasets.NYT)
+word2id = dict((v, idx) for idx, v in enumerate(vocab))
+
+print(X.shape)
+
+print(X.sum())
+# Normal LDA without seeding
+model = guidedlda.GuidedLDA(n_topics=5, n_iter=100, random_state=7, refresh=20)
+model.fit(X)
+
+topic_word = model.topic_word_
+n_top_words = 8
+for i, topic_dist in enumerate(topic_word):
+    topic_words = np.array(vocab)[np.argsort(topic_dist)][:-(n_top_words+1):-1]
+    print('Topic {}: {}'.format(i, ' '.join(topic_words)))
+
+
+# Guided LDA with seed topics.
+seed_topic_list = [['game', 'team', 'win', 'player', 'season', 'second', 'victory'],
+                   ['percent', 'company', 'market', 'price', 'sell', 'business', 'stock', 'share'],
+                   ['music', 'write', 'art', 'book', 'world', 'film'],
+                   ['political', 'government', 'leader', 'official', 'state', 'country', 'american', 'case', 'law', 'police', 'charge', 'officer', 'kill', 'arrest', 'lawyer']]
+
+model = guidedlda.GuidedLDA(n_topics=5, n_iter=100, random_state=7, refresh=20)
+
+seed_topics = {}
+for t_id, st in enumerate(seed_topic_list):
+    for word in st:
+        seed_topics[word2id[word]] = t_id
+
+model.fit(X, seed_topics=seed_topics, seed_confidence=0.15)
+
+n_top_words = 10
+topic_word = model.topic_word_
+for i, topic_dist in enumerate(topic_word):
+    topic_words = np.array(vocab)[np.argsort(topic_dist)][:-(n_top_words+1):-1]
+    print('Topic {}: {}'.format(i, ' '.join(topic_words)))