Obtaining correct core-ids on NUMA shared memory machines

On NUMA architectures it is sometimes cumbersome to obtain the correct core-ids, which are needed for efficient pinning of processes. For example on the multi-socket Westmere EX systems I’m currently working on, it’s not simply 0-9 for the physical core ids of the first CPU but, depending on the machine, sometimes 0-9 scattered over all CPUs or scattered in blocks. On one of the systems with 40 physical cores (80 threads) there was one of the cores numbered with ids 64 and 72, meaning you wouldn’t run on all physical cores if pinning your program to cores 0-39.

There are tools that allow to investigate the memory hierarchy and ids in all details, like Likwid, but on GNU/Linux you can actually derive most required information from /proc/cpuinfo. Hence, I googled a bit around and found this thread, where some of the required work was done already. I modified the script a bit to suit my needs and what came around was that:

[bash]
#!/bin/bash

# ncores
#
# "THE BEER-WARE LICENSE" (Revision 42):
# <br _at_ re-web _dot_ eu> wrote this file. As long as you retain this notice
# you can do whatever you want with this stuff. If we meet some day, and you
# think this stuff is worth it, you can buy me a beer in return.
#
# This script helps to investigate the number of physically available cores on
# a multicore machine and extracts the physical core ids ordered by memory
# affinity (i.e., all cores of one NUMA entity). It is particularly useful
# to do pinning on NUMA machines.
#
# It is based on the script from this thread:
# http://www.dslreports.com/forum/r20892665-Help-test-Linux-core-count-script
#

if [ $1 == "-h" ]; then
echo
echo " Usage: ncores [-h | -l | -p | -v | -n <#proc>]"
echo
echo "Parameters:"
echo
echo " -h Print this help message"
echo
echo " -l Print a list of all physical cores as"
echo " ‘<socket id>,<local core id>’"
echo
echo " -p Print a comma separated list of all physical core ids"
echo
echo " -v Print a comma separated list of all virtual core ids"
echo
echo " -n <#proc> Print #proc core ids as comma separated list, starting"
echo " with physical ids and followed by virtual ids"

exit 0
fi

file="/proc/cpuinfo"

# find the number of real cores/cpus in a box – ignore hyperthreading "logical" CPUs

# check if the physical id and core id fields are there. if not, just use
# raw processor count as the number
num_cores=`grep ‘physical id’ $file | sort -u | wc -l`

if [ $num_cores -eq 0 ]; then
# this box is either an old SMP or single-CPU box, so count the # of processors
num_cores=`grep ‘^processor’ $file | sort -u | wc -l`
list=(`grep -iE ‘^processor’ $file | cut -d: -f2 | tr -d ‘ ‘`)
plist=$list
vlist=()
clist=$list
else
# have to factor in physical id (physical CPU) and core id (multi-core)
# for each "processor" in $file
# concatenate physical_id and core_id then find the unique list of these
# to get the # of cores/cpus
processor_ids=(`grep -iE ‘^processor’ $file | cut -d: -f2 | tr -d ‘ ‘`)
physical_ids=(`grep -iE ‘physical.*id’ $file | cut -d: -f2 | tr -d ‘ ‘`)
core_ids=(`grep -iE ‘core.*id’ $file | cut -d: -f2 | tr -d ‘ ‘`)
list=()
plist=()
index=0
for ent in ${physical_ids[@]}; do
if [ -z "$core_ids" ]; then
list+=( "$ent,-" )
entry="$(printf %04d $ent),0000"
else
core_id=${core_ids[$index]}
list+=( "$ent,$core_id" )
entry="$(printf %04d $ent),$(printf %04d $core_id)"
fi
processor_id=${processor_ids[$index]}
plist+=( "$entry:$processor_id" )
index=$(($index+1))
done

# count number of physical cores
num_cores=`echo ${list[*]} | tr ‘ ‘ ‘n’ | sort -u | wc -l`

# create list of sockets with logical number of cores within sockets
list=`echo ${list[*]}`

# create list of all (physical and virtual) core ids
clist=`echo ${plist[*]} | tr ‘ ‘ ‘n’ | tr ‘:’ ‘ ‘ | sort | cut -c11-`
clist=( $clist )

# create list of core ids with a single id per physical core
plist=`echo ${plist[*]} | tr ‘ ‘ ‘n’ | tr ‘:’ ‘ ‘ | sort | uniq -w 9 | cut -c11-`
plist=( $plist )

# build list of virtual ids as diff of clist and plist
vlist=()
for i in ${clist[@]}; do
skip=
for j in ${plist[@]}; do
[[ $i == $j ]] && { skip=1; break; }
done
[[ -n $skip ]] || vlist+=( $i )
done

# build list of all ids in correct order
clist=( ${plist[*]} ${vlist[*]} )
fi

# request detailed list
if [ "$1" == "-l" ]; then
echo $list | tr ‘ ‘ ‘n’ | sort -u
# request physical id list
elif [ "$1" == "-p" ]; then
echo ${plist[*]} | tr ‘ ‘ ‘,’
# request virtual id list
elif [ "$1" == "-v" ]; then
echo ${vlist[*]} | tr ‘ ‘ ‘,’
#request certain number of ids
elif [ "$1" == "-n" ]; then
[[ $2 -gt 0 ]] && { num_cores=$2; }
echo ${clist[@]:0:$num_cores} | tr ‘ ‘ ‘,’
# print only number of cores
else
echo $num_cores
fi

exit 0
[/bash]

If called without any parameters, it just prints the number of physical cores. However, what I usually use for pinning, is a call like ncores -n 10, to obtain the first ten core ids such that they are placed as close to each other as possible, hence reducing memory access latency. To automatically set the required environment variables for OpenMP, I embedded it in a script:

[bash]
#!/bin/bash

if [[ $# -lt 2 ]]; then
echo "Usage: $0 <np> <binary> <arg1> <arg2> …"
exit 1
fi

NPROCS=$1
PROCLIST=$(ncores -n $NPROCS)

KMP_AFFINITY="granularity=fine,proclist=[$PROCLIST],explicit"
GOMP_CPU_AFFINITY="$PROCLIST"
OMP_NUM_THREADS=$NPROCS

export KMP_AFFINITY OMP_NUM_THREADS GOMP_CPU_AFFINITY

echo "Setting KMP_AFFINITY to ‘$KMP_AFFINITY’"
echo "Setting OMP_NUM_THREADS to ‘$OMP_NUM_THREADS’"
echo "Setting GOMP_CPU_AFFINITY to ‘$GOMP_CPU_AFFINITY’"

shift
BINARY="$1"
shift

${BINARY} "$@"
[/bash]

Deployment of FeedHQ on Apache 2.2 – an Open Source Google Reader alternative

Click here to get directly to the deployment tutorial.

When Google announced last year that they’re discontinuing Google Reader, this came like a shock to many people. Especially since it seems like there were only few alternatives available, of which even less provided similar functionalities as Google did. Especially when it comes to synchronization services, that allow keeping feeds up-to-date across multiple devices, there were no real options other than Google Reader.

However, a lot has happened in terms of developments since that date: Some services, like Feedly or Digg, attempted to take over. Especially Feedly gained a lot of new users after the shutdown announcement. For those comfortable to host a synchronization service by themselves Tiny Tiny RSS became a popular choice. And an attempt to define a common and open synchronization interface, called Open Reader API, was born – widely based on the Google Reader API – and is (so far) implemented by two news aggregators: BazQux and FeedHQ.

I haven’t been using Google Reader or any other news aggregator yet but simply relied on my beloved Vienna, a very good OpenSource RSS client for OS X. But as a fresh owner of a new Fairphone (instead of an old shitty iPhone 3G S) I now wanted to be able to read the feeds also on other devices and online on a web interface. As Vienna supports the Open Reader API it was a natural choice to go for that solution.

FeedHQ and BazQux both provide extremely cheap accounts (12$/year or 9-29$/year, respectively), so it would have been an easy choice to just sign up there and use their services. But I’m a fan of keeping my data with me and hosting as much on my own server as possible (even though both provider state that they’re dedicated to users’ privacy), as I also do it for contacts, calendars and file sharing with my own instance of OwnCloud (instead of Google Calendar) and my photos on my own Gallery (instead of using flickr or something similar). And so far I’ve always felt reassured towards this attitude, for example recently when Feedly decided over night to only allow Google+ logins.

Therefore I chose the great FeedHQ-project, which is available as Open Source Software from GitHub.

Installing FeedHQ on Apache 2.2

Getting FeedHQ to work is not very difficult – but you should know what you’re doing. However, in combination with the popular Apache webserver (at the time of writing I’m using Apache 2.2.22 on Ubuntu 12.04 LTS) there a few small tricks one needs to know about, to get it working properly. I figured them out in a long debug-session with FeedHQ’s developer Bruno Renié.

FeedHQ is based on the Django Python Web-Framework, so if you have deployed or even developed an application with that framework before, you probably know about most of the following steps. A short installation manual is also available.

1. Install required dependencies
At the moment these are Python 2.7, Redis (2.6+ recommended) and PostgreSQL (9.2+ recommended). As I’m using most of such packages from the Ubuntu repositories, I have to rely on the versions available there, which are Python 2.7.3 and PostgreSQL 9.1.11. However, Redis is hopelessly outdated in the repositories, so I had to install that one manually (for example as described here).

If you want to use WSGI for the deployment (like I did), you also need the package libapache2-mod-wsgi from the Ubuntu repositories.

2. Check out the current version of FeedHQ
Obviously, the first step with FeedHQ is to obtain the current version of the code from GitHub:

[bash]
git clone https://github.com/feedhq/feedhq.git
cd feedhq
[/bash]

3. Setup virtual python environment and install dependencies
This is done with the following commands. You might have to install python-virtualenv, python-pip and virtualenvwrapper from the repositories first.

[bash]
virtualenv -p python2 env
source env/bin/activate
add2virtualenv .
pip install -r requirements.txt
[/bash]

4. Setup environment variables

FeedHQ relies on a number of environment variables, which are described in the configuration section. Unfortunately, WSGI does not pass any environment variables from the Apache configuration as os.environ['VAR_NAME'] on to the wsgi-script. There are a few very ugly workarounds for that, but none of them really convinced me. Therefore I just hard-coded them directly into feedhq/wsgi.py like that:
[python]
os.environ[‘DJANGO_SETTINGS_MODULE’] = ‘feedhq.settings’
os.environ[‘SECRET_KEY’] = ‘some very long secret key’
os.environ[‘ALLOWED_HOSTS’] = ‘sub.domain.com’
os.environ[‘FROM_EMAIL’] = ‘feedhq@domain.com’
os.environ[‘REDIS_URL’] = ‘redis://localhost:6379/1’ # (or however your redis is configured)
os.environ[‘DATABASE_URL’] = ‘postgres://<user>:<password>@localhost:5432/feedhq’ # (or however your database is set up)
os.environ[‘HTTPS’] = ‘1’ # (if you’re going to use an HTTPS connection)
[/python]

It’s quite ugly, I know, but it was a fast fix. If you know of any more elegant solution, please let me know!

A short version of the corresponding configuration for the virtual host is (only the most important entries and assuming, feedhq was checked out to /var/www/feedhq):
[bash]
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName feedhq.domain/com
SSLEngine On

# Static files, media and css
AliasMatch ^/([^/]*.css) /var/www/feedhq/feedhq/static/styles/$1
Alias /static/ /var/www/feedhq/feedhq/static/
Alias /media/ /var/www/feedhq/feedhq/media/

<Directory /var/www/feedhq/static>
Order deny,allow
Allow from all
</Directory>

<Directory /var/www/feedhq/media>
Order deny,allow
Allow from all
</Directory>

# Let Django handle the authorization, not Apache
WSGIPassAuthorization On

# Allow encoded slashs
AllowEncodedSlashes On

# WSGI script
WSGIScriptAlias / /var/www/feedhq/feedhq/wsgi.py

<Directory /var/www/feedhq/feedhq>
<Files wsgi.py>
Order allow,deny
Allow from all
</Files>
</Directory>
</VirtualHost>
</IfModule>
[/bash]

5. Create admin user

Finally, you just need to create an admin user to get started:

[bash]
manage.py createsuperuser –username=joe –email=joe@example.com
[/bash]

Using that one you can now login to the admin panel feedhq.domain.com/admin, setup more users, groups and get to know FeedHQ.

6. Setup Cron-Jobs and task queue

To get all feeds and maintenance tasks done, a number of cronjobs is necessary, which again rely on a few workers that have to be running.

All required cronjobs are described in the installation manual. To setup the environment correspondingly, I’ve encapsulated the commands in a small bash-script, that first sets the variables and then executes the requested command.

To make sure that the workers are always online when restarting the server, I’ve written a small upstart script (since Ubuntu does now support upstart) that’s stored as /etc/init/feedhq:
[bash]
# FeedHQ
#
# Server Application with OpenReader API to synchronize RSS Feeds

description "FeedHQ"

start on (postgresql-started and runlevel [2345])
stop on runlevel [!2345]
respawn

exec start-stop-daemon –start -c www-data –exec /scripts/www-data/feedhq_run_django-admin.sh rqworker store high default favicons
[/bash]

As you see, it does not directly call the django-admin but instead the before mentioned script, which only sets the appropriate environment variables first. It also waits that PostgreSQL is started first (since the DB is required), but unfortunately can’t wait for redis without too much of a trouble. Therefore I skipped this check for now.

With cronjobs and workers set up you’re done and can start enjoying FeedHQ.