Strange issue with exported .odt file and Caliber

[mogplus88]

Hi All,

I have been experimenting recently with using Caliber to create an epub version of my book to upload to Kindle. To do this I created a .odt file from NW, then added it to Caliber and attempted a conversion to epub, which failed. The error seemed to be related to a problem with font family. Tried a lot of mucking around with fonts but to no avail.

So, in an attempt to find out where it was failing I decided to copy the first half of the document generated by NW into a new file and try to convert that. Success! So I tried copying more of the file, thinking that I would eventually reach a point where it failed and I could identify the point that was causing the problem. It didn't happen. I eventually copied the whole file into a new file, and the file converted successfully.

So I created a much smaller file (using the NW example project) and tried the same thing. The generated file didn't work in Caliber, but a copied file did.

Strange I thought, as I noticed a couple of other weirdnesses too. 1. The file I exported from NW used the font Liberation Sans 10pt to create the output file, and that is the font that was in the output file. But when I copied it, the new version used Liberation Serif 12pt instead. 2. The copied file was much bigger. The file NW created was ~16.1k but the copied version was ~25.3k. However both files appeared to be identical when opened with Libreoffice Writer. I didn't check it word for word but a quick scan comparing the two showed no text discrepancies. Changing the font back to Liberation Sans 10pt in the copied file made no difference to the size. 3. The heading in the NW generated document does not appear in the copied document.

I initially started searching in a Caliber forum but found absolutely nothing even vaguely related to this problem, so I'm posting this here in case the original problem, that Caliber rejected the original file created by NW for conversion, could be related to how it is generated by NW.

But why a different version copied and pasted seems to produce a radically different file is a mystery to me. I'm not even sure where I could ask the question. Maybe a Linux forum? I'll try that.

:-\ Ian

EDIT. I thought it must be related to Libreoffice Writer. I've posted a question in their forum.

[vkbo]

I know that AbiWord crashes (or at least used to crash) when opening a novelWriter generated ODT file, but not if you open and save it again in LibreOffice. I never managed to find out why, as AbiWord crashes with no error. The novelWriter ODT file passes validation in the OpenDocument validator tool, so I'm not sure what the cause could be. I suspect AbiWord assumes there is content there that LibreOffice and OpenOffice adds, but that novelWriter does not. Maybe it's the same here.

When novelWriter generates an ODT file, it only writes the info it must, to be compliant with the standard. LibreOffice writes a lot more info into the file, like a ton of settings that are specific to LibreOffice. My assumption was that the other applications would just add their defaults on their own. I think I also enabled compression on the ODT file in novelWriter. Not all office applications do that (they're just Zip files).

In any case, none of this has anything to do with your text inside the document I expect, but more to do with all the other information that is added under the hood. Like how font is specified. When I wrote the ODT export format, I looked at how font is formatted.

What you could try is to export fodt instead of odt, and do the same in LibreOffice. The fodt file is a plain XML file, and it is possible to compare those in a plain text editor. You'll notice that the LibreOffice version has a lot more info aside from your text than the novelWriter file has. If you can figure out exactly what's causing the issue, I'm happy to try to fix it at my end. I can probably have a look myself as I have Calibre installed.

[peter88213]

That could indeed be an indication that Calibre may not support the version of the ODT format generated by novelWriter. In LibreOffice, you can specify which file format version to save in. I’ve set mine to a lower version to ensure compatibility with OpenOffice 3.4, which is what I mainly use on a daily basis.

[vkbo]

I can look into downgrading the format to 1.2 instead. novelWriter only uses a couple of settings from the 1.3 standard.

[peter88213]

That would help OpenOffice users, and might also improve compatibility with other word processors.
However, that doesn’t necessarily mean the problem with Calibre would actually be solved. But if I recall correctly, you already have an EPUB export feature in the pipeline, don’t you?

[vkbo]

Yes, I do. It is mostly finished, but I stopped working on it because I am uncertain how to handle project notes in the context of the Epub format. Should I just block them on export, or allow the user to just export them if they select them? How do I treat them in terms of chapters then? Because an ereader would consider them chapters.

[peter88213]

I have to admit, the discussion here—and the realization that the best EPUB exporter for LibreOffice is no longer officially available—has inspired me to program an EPUB export feature for my novel organizer as well. I wouldn't export any metadata; the users can organize whatever they want to include in their e-book as regular chapters.

[peter88213]

@mogplus88 I also suspect that Calibre analyzes some data that is specifically written by LibreOffice or OpenOffice, but which may be optional by the standard. That doesn't matter, though, because an Office text document contains far too much formatting information for an EPUB anyway—information you'd probably have to painstakingly pick out yourself with Calibre. Newer versions of LibreOffice include built-in EPUB export, and there’s also a very good extension called writer2epub that, in my opinion, creates even better EPUB files. Have you tried it yet? You could use these EPUB documents as a starting point for fine-tuning with Calibre, which saves a lot of cleanup work.

Apart from that, I would simply open the ODT file exported from novelWriter in LibreOffice and save it again, as suggested above. You could also apply a document template at this point to customize the layout to your liking.

[mogplus88]

@peter88213 thanks for the tips. I have tried exporting an epub from Writer, but I suspect (based on no evidence other than pure paranoia) that it would not do as good a job as Caliber, as exporting an epub is a small and probably little used feature of Writer, so maybe not fully tested and verified. Whereas it is Caliber's raison d'etre. Another slight concern I have is that when I have imported the epub from Writer into Sigil it detects (and automatically corrects) a number of errors.

I haven't tried writer2epub but I will certainly give it a shot.

EDIT. Seems writer2epub only works on Windows. I use Linux... :-(

;-) Ian

[peter88213]

The thing is, Office text documents usually contain a lot of information related to the print layout, which is simply unnecessary in an EPUB, since the reader handles the presentation. Some EPUB converters attempt to produce a “faithful” layout for the EPUB, which results in an overload of styles and generally causes problems with different readers and screen sizes. The key feature of the LibreOffice EPUB exporters is their ability to map the document structure and filter out print-specific data. The writer2epub plugin generates a simple and clear HTML structure that is also easy to edit manually.
If Sigil corrects “errors” during the process, you should try to understand exactly what the criteria are. In general, it’s worth building up some know-how if you want to create EPUBs for self-publishing on your own. I’ve been looking into this for a while, and I definitely prefer building a simple EPUB document step by step until it’s “ready for sale” rather than trying to clean up the mess in an Office document.

[peter88213]

Okay, I just checked: There’s a different writer2epub extension available for LibreOffice than the one I’m using here with Apache OpenOffice. This one is actually platform-independent. The extension from the OpenOffice download page also works for me with LibreOffice 7:
https://extensions.openoffice.org/de/project/writer2epub.html

Update

Upon closer investigation, it becomes clear that the LibreOffice Extensions site offers an extension called WriterToEPUB, which appears to include a Windows executable. The extension I’m referring to is called writer2epub and is actually only available on the Apache OpenOffice Extensions site.

2nd update

Out of curiosity, I just compared the EPUB export with LibreOffice 7.6.7:

writer2epub extension

• One XHTML file per chapter.
• A proper table of contents.
• A set of quite generic stylesheets.
• Custom styles from the ODT document are replaced with generic styles.

The built-in EPUB exporter

• An incomplete table of contents (only the first chapter is listed).
• The whole content is put into a single XHTML file.
• A set of stylesheets that reflect the document's settings, such as sizes and spacings in printer units.

I'm using a rather outdated LibreOffice version; perhaps the EPUB export has been improved in the meantime. However, Linux distros might come with even older LibreOffice versions, so it's advisable to install the writer2epub extension anyway.

[mogplus88]

Had a look at the link, the site warns that the extension hasn't been update for a while (the last update was 2014) and the download is not available any more. It's just a blank page. A search for apache openoffice writer2epub takes me to the same page.

Anyway, probably not worth pursuing really. So I've been experimenting with a completely different approach, exporting the file from NW as a markdown file. Caliber is quite happy to import the file and convert it to epub, with only one tiny fly in the ointment that I have stumbled across (so far...). This is because I've defined the heading for a scene to be replaced by "* * *", a fairly common thing to do. But unfortunately, along the way somewhere, the string is interpreted as markdown, and turned into an unordered list. I've also tried a short string of underscores, but they get changed to a full width horizontal bar. Not really appropriate for a scene change.

Rather than pursuing it here I'm thinking of posting a general question on the forum asking how other NW users convert their documents to epub. Why reinvent the wheel?

[peter88213]

Yes, I've since found out that the website of the author of writer2epub is down. His blog, which he lists as a contact and alternative download option, is also down. Everything points to the fact that he has stepped back and abandoned his extension. As a result, writer2epub shares the fate of other excellent OpenOffice extensions, such as Pepito Cleaner, which still work flawlessly today despite a lack of maintenance. Unfortunately, there is also no information regarding any license that would allow me to make the extension available for download. Nowadays, thanks to open-source platforms like GitHub and the widespread use of licenses, it is much easier to make free software available over the long term.

That said, I think your approach using Markdown is excellent. It helps you avoid problems with unnecessary formatting right from the start and gives you the chance to end up with a very clean ebook. The issue with the asterisks should be easy to fix: they only count as bullet points if they appear at the beginning of a line. You could put a space before them or fence them with backticks, like so: `* * *`.
Another approach would be to replace them with neutral special characters and then undo the whole thing later in Calibre using a global search and replace. However, it wouldn’t hurt to take a look at the Markdown specifications to gain some certainty here.

Update

I forgot to mention my own solution for section dividers: I define them as third- or fourth-level headings, which allows me to easily format them using a stylesheet—that is, to center them and add a line-height of one line above and below each one. Of course, you then have to make sure they don’t appear in the table of contents, so you need to limit the number of levels there.

[mogplus88]

Tried another approach today, which seems to work as well. Exporting from NW in .html format. Caliber can open it and the formatting from NW seems to be maintained. I've converted my book to AZW3 (Kindle format) and so far it looks good. The TOC looks a bit strange, as Calibre seems to have randomly selected three level two headings to include in it (from three non-consecutive chapters), and nothing else...

;-)

[vkbo]

Strange. Have you formatted you chapters with ## Title headings? It should export those as <h1> in the HTML. You can verify by opening the HTML document in a browser.

[vkbo]

But yes, the HTML format is intended for exactly this use. It's an almost universally supported rich text format these days.

[mogplus88]

Caliber seems to like it. The TOC weirdness has disappeared, not sure what happened there. I was mucking around a lot with headings and separators, so probably a problem between keyboard and seat. Anyway it's looking good now. And easy too. I just import the .html file into Calibre, add the cover page image to the metadata, and click the convert button. I've emailed the latest version to my Kindle, so I'm off to see what it looks like on an actual Kindle! (Actually a vintage Lenovo tablet with the Kindle app, which is awful, but I'm too tight to buy an actual Kindle.)

;-)

[mogplus88]

Well, that was interesting. Turns out AZW3 is not a valid format to upload to Kindle by email. So I have no idea how to get it onto my Kindle. A bit of searching told me that Amazon will convert an uploaded epub book to azw3, so my confusion now lies in why tools like Calibre (and others) would bother converting anything to azw3. You can't upload it to Amazon, but, according to the experts, anything you upload is converted to azw3 anyway. Duzzen madder. SEP.

Let me rephrase that. I haven't discovered a way to upload a azw3 to Amazon. Somebody smartere and/or more knowledeable than me may know how to do it.

So I've got my book on my Kindle now. I've found that reading the book on my Kindle is a very different experience to reading it on the computer. Formatting becomes a much more critical issue it seems. Not to mention all the typos and plot oddities I'm finding...

:-\

[vkbo]

Yeah, there are also variations in the formats for different devices. I use a Kobo, and it does not like CSS styling. I've had to strip it on some books.

[peter88213]

my confusion now lies in why tools like Calibre (and others) would bother converting anything to azw3.

I haven't got a Kindle e-reader, but I use the Kindle app on my smartphone for reading e-books. If I connect my smartphone to the PC via USB, Calibre can mount it and upload e-books. Uploaded AZW3 documents do appear in the Kindle library. So I can read my own work on the phone with no network connection involved.
The same works with EPUB and my e-reader. Did you try it with your Kindle reader?

[peter88213]

I use a Kobo, and it does not like CSS styling.

I've got a Tolino, which appears to be a re-branded Kobo. It renders my self-generated EPUBs almost exactly as specified in my CSS stylesheets. The only exception so far is small caps, which are ignored. In the AZW3 files converted by Calibre, the stylesheet appears to be ignored.

@mogplus88 I just tried the MOBI format, which partially applies the CSS instructions—acceptable overall. And as I can see, the Kindle app on the smartphone also supports the MOBI format. Would be worth a try with your Kindle reader, wouldn't it?

[mogplus88]

@peter88213 I can't upload anything directly to my Kindle, which is actually an old Lenovo tablet with the Kindle app. I've had it for years, and the usb plug is worn out I think. It still works to charge the device, but communication is impossible. When I plug it into my PC it connects and disconnects every second or two. At one point it connected long enough for Calibre to recognise it and ask if I would like it to manage the device. I clicked OK but by that time it had disconnected again. I've tried a few different cables, but none of them work. I've given up.

:-(