Blogger / Blogspot issue

[AndyTheFactory]

Issue by ontopicprojects
Mon Oct 5 15:37:20 2020
Originally opened as codelucas/newspaper#847

---

Some blogspot / blogger sites don't seem to parse: here is an example:

`from newspaper import Article

url = 'http://www.righto.com/2011/07/cells-are-very-fast-and-crowded-places.html'

article = Article(url)
article.download()
article.parse()
print(article.text)`

this prints ""

[AndyTheFactory]

Comment by johnbumgarner
Thu Oct 8 03:50:57 2020

---

The primary reason that you cannot extract from this site with Newspaper is because the tags commonly queried by this module do not exist on the website http://www.righto.com. You should use the Python libraries requests and BeautifulSoup to extract the items that you want.

[AndyTheFactory]

Comment by AndyTheFactory
Tue Oct 4 16:28:18 2022

---

you can achieve this with some minor changes.

1. the cleaners class must be fixed (it removes itemprop containing articleBody if it is not exactly "articleBody") see fix itemprop containing articleBody codelucas/newspaper#953
2. extend the extractor class and replace it in your article 👍

class myExtractor(ContentExtractor):
    def nodes_to_check(self, doc):
        generator = self.parser.getElementsByTag(doc, tag='meta', attr={'name':'generator'})
        for t in generator:
            if t.attrib.get('content') == 'blogger':
                nodes = self.parser.getElementsByTag(doc, tag='div', attr={'class':'post-body'})
                return nodes
        return super().nodes_to_check(doc)

and

a = Article('http://www.righto.com/2011/07/cells-are-very-fast-and-crowded-places.html')
a.extractor = myExtractor(a.config)
a.download()
a.parse()

[AndyTheFactory]

Works in v0.9.1