Trondheim

Vergangene Woche haben wir das Brückentags-Wochenende genutzt, um eine Freundin in der norwegischen Stadt Trondheim zu besuchen. Das im Vergleich zu Stockholm kleine Städtchen (ca. 180 000 Einwohner) hat uns sofort mit seiner malerischen Lage am Trondheimsfjord zwischen den Bergen und dem gemütlichen Stadtzentrum richtig gut gefallen.

Das wechselhafte Wetter hat uns immerhin an ein paar Tagen richtig schönene Sonnenschein geschenkt, so dass wir gleich am ersten Tag gemütlich in der Sonne sitzen und am Samstag eine Wanderung mit Grillen am See machen konnten. Durchaus glücklich, denn eine Woche zuvor hatte es nochmals geschneit. Zeitgleich war auch noch Jazz-Festival in der Stadt mit einer ganzen Reihe toller und noch dazu kostenloser Konzerte.

Beeindruckend war auch das Preisniveau: Schon in Schweden kann man bei dem meisten ja nicht gerade von Schnäppchen sprechen – aber multipliziert man den schwedischen Kronenbetrag mit 1,5 erhält man den norwegischen Kronenbetrag. Dazu kommt dann noch der teurere Wechselkurs und schon rinnt einem das Geld nur so durch die Finger… Eine Dose lokales Bier im Supermarkt kostet z. B. 28 NOK, umgerechnet ca. 3,75 €, und das Maxi-Menü bei der Fastfood-Kette mit dem goldenen Buchstaben ganze 97 NOK bzw. knapp 13 €.

Dafür merkt man an anderen Stellen den Reichtum des Landes und dass einfach viel Geld für Sachen zur Verfügung steht, die anderswo knapp oder gar nicht finanziert werden. Dazu zählt nicht nur das tolle Jazz-Festival, auch viele Studenten- und Schülerrabatte sind prozentual deutlich größer als etwa in Deutschland und so richtig beeindruckt waren wir von den prächtigen und massiven Grillplätzen an den Seen.

Beeindruckend war aber auch schon der merkbar längere Tag. Am Freitag haben wir den Sonnenuntergang bei einem (dreistündigen) Spaziergang am Ufer genossen. Gemütlich nach dem Abendessen – die Sonne verschwand ja erst um 22.00 Uhr hinter den Bergen. Tageslicht bzw. Dämmerung war dann locker noch bis kurz vor Mitternacht. Bedenkt man, dass noch über einen Monat lang die Tageslänge zunimmt, darf man dort wirklich keine Probleme mit dem Schlafen bei Tageslicht haben…

Bilder gibt es natürlich wie immer in der Gallery.

LAMPP+Calibre+OwnCloud = my own fully integrated online-library and news-aggregator

Inspired by the recent Twitter-campaign #ichwillzahlen I thought, what I would like to have right to read text news. In my case this is a collection of several news feeds, that I can read on my Kindle during breakfast or on the way to university. But there is no handy and comfortable solution available, yet. So, I decided to create it myself.

My wish list of features:

  • Aggregation of several online medias (homepages of newspapers, news channels etc.)
  • Automaticpreparation in an e-book-reader friendly format (for the Kindle, this is .mobi)
  • Automatic download/transmission to my Kindle

If you just want to have a look what is now possible: have a look at the conclusion.

Status quo

I had a few solutions tested some time ago, and they all were too complicated/time consuming in the morning: Calibre offers a proper way to manage e-books and sync them with a Kindle. It also has a plugin readily available, that allows to download news feeds with a lot of presets and put them in a nice format. But to make it work automatically, my Notebook has to run including Calibre. Then it downloads feeds automatically and even sends them to the Kindle-mail-address at Amazon.

Nowadays everything goes to the cloud, why not also this?

Calibre offers a nice feature, a content server, that allows to browse the library in HTML-format and download ebooks. So, lets put Calibre on the server and make my library available there. But I would like to still maintain and fill it on my local notebook.

First step: sync Calibre Library with OwnCloud

Since the Calibre Library is just a folder, that contains many subfolders with the ebooks and some metadata plus a sqlite-database, it is quite easy to sync it. The most comfortable way for me was to use my existing installation of OwnCloud to do that. I just added the folder to the list of Sync-Folders – and voila: it’s as easy as Dropbox. Since OwnCloud (un)fortunately doesn’t offer encryption of the files on the server, the whole library is now available on the server.

Second step: Calibre Content Server

To make the library accessible, I had to install Calibre on the server. Thanks to out-of-the-box usable packages this is no problem at all. Luckily the content server integrates neatly with existing webservers using WSGI, so it’s as easy as modifying the given script with the correct paths and updating the virtual host of ownCloud, to provide the content server under the folder ‘/calibre’. My WSGI-script looks like that, with all paths suitable for Ubuntu 12.04:

[code language=”python”]
# WSGI script file to run calibre content server as a WSGI app

import sys, os

# You can get the paths referenced here by running
# calibre-debug –paths
# on your server

# The first entry from CALIBRE_PYTHON_PATH
sys.path.insert(0, ‘/usr/lib/calibre’)

# CALIBRE_RESOURCES_PATH
sys.resources_location = ‘/usr/share/calibre’

# CALIBRE_EXTENSIONS_PATH
sys.extensions_location = ‘/usr/lib/calibre/calibre/plugins’

# Path to directory containing calibre executables
sys.executables_location = ‘/usr/bin’

# Path to a directory for which the server has read/write permissions
# calibre config will be stored here
os.environ[‘CALIBRE_CONFIG_DIRECTORY’] = ”

del sys
del os

from calibre.library.server.main import create_wsgi_app
application = create_wsgi_app(
# The mount point of this WSGI application (i.e. the first argument to
# the WSGIScriptAlias directive). Set to empty string is mounted at /
prefix=’/calibre’,

# Path to the calibre library to be served
# The server process must have write permission for all files/dirs
# in this directory or BAD things will happen
path_to_library=’/data//files/’,

# The virtual library (restriction) to be used when serving this
# library.
# virtual_library=None
)

del create_wsgi_app
[/code]

only the path_to_library has to modified, depending on your path to OwnCloud, Username in OwnCloud and name of the synced Library folder. To restrict access to myself I added simple htaccess-authentication to the Content-Server. Now it is already possible to browse the library online and download ebooks. This works also with the (experimental) browser provided by the Kindle itself. Important: due to a bug in Python or Calibre (noone feels responsible), the content server has to be accessed using ‘/calibre/’, i.e. include a trailing slash. Otherwise you will end up with Internal Server Errors.

Step 3: News aggregation

But now I don’t have the latest news on my Kindle yet. Luckily, someone did that before me. A script called news2kindle downloads a list of news feeds using Calibres recipes, packs them into a zip-file and sends them to the Kindle mail address. This worked perfectly fine out of the box. I created two lists with sources, one for daily and one for weekly feeds, and call the script with each list from a cronjob with the respective frequency.

Step 4: Adding news to the library

To complete the combination: I also wanted to add the downloaded newsfeeds to my calibre library, such that I could also download them via the web interface. Calibre allows to add files to its library using the command ‘calibredb add’. This will copy the ebook to the library folder and insert it into the database. Simply adding that to news2kindle makes the news now also available via the content server. But: OwnCloud has a caching technique for the list of files stored. And it doesn’t know anything about those newly added ebooks. Unfortunately, OC in version 5.0.x doesn’t provide any functionality to rescan the file tree and it seems the developer are also not aiming on adding it. Does that cause a problem? It does indeed! When syncing the library back to my notebook, the Calibre database will know about the ebooks from the news feeds, but won’t have the respective files. So how to make OC aware of those?

The cleanest solution would be to simply sync the Library from OwnCloud to a dedicated folder on the webserver. This would cause to have it available there twice, but other than that everything works fine. Unfortunately, the Sync Client needs an XServer to run. So I first tried this alternative solution which I simply didn’t get to work.

So instead I used the webdav mount as a quick-and-dirty fix. Mounting the Calibre-Library as a webdav-resource directly from OwnCloud makes it aware of all file-changes. However, calling ‘calibredb add’ on this mount is not possible, since the command modifies an sqlite database, which is not possible on webdav resources without further tweaks. But I do know (or can influence) where the new ebooks will be stored in the library folder (which is simply the folder ‘calibre’ in the Library folder). So I still call ‘calibredb add’ directly as before and just touch everything in calibre on the webdav mount. That way OC knows about the new files. Of course the Webdav-mount has to be configured appropriately in fstab. My resulting news2kindle script looks like that:

[code language=”bash”]
#!/bin/bash
# —————————————————————————–
# news2kindle.sh
# Copyright (C) 2011 Gerald Backmeister (http://mamu.backmeister.name)
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see .
# —————————————————————————–
# Version: 0.1

if [[ $# -lt 1 ]]; then
echo "No recipe file given!"
exit 1
fi

recipes_file=$1

# If news shall be added to a library: add the path here, otherwise leave it empty
CALIBRE_LIBRARY_WEBDAV=""
CALIBRE_LIBRARY="/data//files/"

NEWS_DIR="/tmp/news2kindle"
CALIBRE_RECIPES_DIR=/usr/share/calibre/recipes
TARGET_FORMAT_EXTENSION=".mobi"
MAIL_RECIPIENT="@kindle.com"
MAIL_SENDER=""

ERRORS=0

sendmail() {
mail_recipient=$1
message=$2
subject=$message
attachment_file=$3
if [[ -z $4 ]]; then
flags=""
else
flags="-f$4"
fi
(
echo "To: $mail_recipient"
echo "Subject: $subject"
echo "MIME-Version: 1.0"
echo ‘Content-Type: multipart/mixed; boundary="-q1w2e3r4t5"’
echo
echo ‘—q1w2e3r4t5’
echo "Content-Type: text/html"
echo "Content-Disposition: inline"
echo $message
echo ‘—q1w2e3r4t5’
echo ‘Content-Type: application; name="’$(basename $attachment_file)’"’
echo "Content-Transfer-Encoding: base64"
echo ‘Content-Disposition: attachment; filename="’$(basename $attachment_file)’"’
/usr/bin/uuencode –base64 $attachment_file $attachment_file
echo ‘—q1w2e3r4t5–‘
) | /usr/sbin/sendmail $flags $mail_recipient
return $?
}

NEWS_SUB_DIR=`date +"%Y%m%d_%H%M"`

# mount webdav folder
mount "$CALIBRE_LIBRARY_WEBDAV"
ERRORS=$((ERRORS+$?))

# Create folder
mkdir -p "$NEWS_DIR/$NEWS_SUB_DIR"
ERRORS=$((ERRORS+$?))

recipe_counter=0
while read recipe; do
if [[ $recipe == #* ]]; then
continue
fi
if [[ $recipe == "" ]]; then
continue
fi

if [[ $recipe == */* ]]; then
recipe_file="$recipe"
else
recipe_file="$CALIBRE_RECIPES_DIR/$recipe.recipe"
fi
recipe_name=`basename $recipe_file .recipe`
target_base="${recipe_counter}_${recipe_name}"
recipe_target_file="$NEWS_DIR/$NEWS_SUB_DIR/${target_base}${TARGET_FORMAT_EXTENSION}"
recipe_log_file="$NEWS_DIR/$NEWS_SUB_DIR/$target_base.log"
echo "Downloading $recipe_name to $recipe_target_file…" >"$recipe_log_file"
/usr/bin/ebook-convert "$recipe_file" "$recipe_target_file" >>"$recipe_log_file" 2>>"$recipe_log_file"
ERRORS=$((ERRORS+$?))
chmod +r "$recipe_target_file"
if [[ -n $CALIBRE_LIBRARY ]]; then
/usr/bin/calibredb add -d –with-library "$CALIBRE_LIBRARY" "$recipe_target_file"
ERRORS=$((ERRORS+$?))
touch "$CALIBRE_LIBRARY_WEBDAV/calibre/*"
fi
recipe_counter=`expr $recipe_counter + 1`
done < $recipes_file # unmount webdav folder umount "$CALIBRE_LIBRARY_WEBDAV" ERRORS=$((ERRORS+$?)) cd "$NEWS_DIR/$NEWS_SUB_DIR" >/dev/null

echo "Compressing documents…"
zip_file=news2kindle.zip
/usr/bin/zip $zip_file *${TARGET_FORMAT_EXTENSION}
ERRORS=$((ERRORS+$?))

echo "Sending email…"
sendmail $MAIL_RECIPIENT news2kindle $zip_file $MAIL_SENDER
ERRORS=$((ERRORS+$?))

cd – >/dev/null

# all fine? then delete the tmp-files
if [[ $ERRORS -gt 0 ]]; then
rm -rf "$NEWS_DIR/$NEWS_SUB_DIR" >/dev/null
ERRORS=$((ERRORS+$?))
fi

# check for errors to set return value
if [[ $ERRORS -gt 0 ]]; then
exit 1
else
exit 0
fi
[/code]

Conclusion

Now my preferred news feeds are downloaded automatically to my Calibre Library and are sent to my Kindle via E-Mail. All I have to do in the morning is to active the WiFi on my Kindle. At the same time I can also access all my ebooks over a (protected) web-interface and download them directly to the e-Reader. No more need to sync them via USB.

One could now also add some functionalities to automatically remove old issues of the news feeds from the library to avoid bloating it. This is also simply possible via ‘calibredb remove’, but requires a few thoughts on how to determine the proper ids of the entries. Since the timestamp of folders of old issues are updated each time new feeds are added, its not possible to identify them this way. Again, for that one would need to make OC aware of the changes to the file system, which is already not solved as elegant as possible.

A word of warning to the end: Having Calibre open during the time the news feeds are updated will lead to conflicts between the databases on the server and your local computer. Be aware of that and avoid it! I solved it for me by downloading the feeds early in the morning, when I’m (more or less sure), that I won’t be awake anyways… Also I recommend to use the same Calibre versions on the server and your local computer. I’m not sure how well different versions are compatible.