GNU Emacs workshop at c-base / Berlin

Emacs? Emacs! I proudly announce the first Emacs workshop held by the incredible Anselm Helbig (also known as „mnemonikk“), starting next Monday, the 22nd of March at c-base/Berlin/Germany. If you ever wanted to start using Emacs as an advanced editor and development environment, this is the right kick off workshop for you. Anselm is ready to answer all your questions regarding installation, configuration, usage and programming of the GNU Emacs editor. The focus of the workshop is a hands on experience rather than a lecture.

A detailed German description of the workshop can be found in Anselm’s blog. The workshop is free of charge.

When backups fail: A mysql binlog race condition

Today I ran into my first MySQL binlog race condition: The initial problem was quite simple: A typical MySQL master->slave setup with heavy load on the master and nearly no load on the slave, which only serves as a hot fallback and job machine, showed differences on the same table on both machines. The differences showed up from time to time: entries that have been deleted from the master were still on the slave.

After several investigations I started examining the MySQL binlog from the master – a file containing all queries that will be transferred to the slave (and executed there if they don’t match any ignore-db-pattern). I grepped for ids of rows that have not been deleted on the slave as I’s interested if the DELETE statement was in the binlog. In order to read a binlog file just use „mysqlbinlog“ and parse the output with grep, less or similar. To my surprise I found the following entries:

$ mysqlbinlog mysql-complete-bin.000335 | grep 1006974
DELETE FROM `tickets` WHERE `id` = 1006974
SET INSERT_ID=1006974/*!*/;

As „SET INSERT_ID“ is a result of an INSERT statement it was clear, that MySQL wrote the INSERT => DELETE statements in the wrong order. As INSERT/DELETE sometimes occur quite fast after each other and several MySQL  threads are open in the same MySQL server, you might run into a rare INSERT/DELETE race condition as the master successfully executes them, while the slave receives them in the wrong order.

As a comparision this is a normal order of INSERT and DELETE (please note that the actual INSERT is not displayed here):

$ mysqlbinlog mysql-complete-bin.000336 | grep 1007729<br />SET INSERT_ID=1007729/*!*/;<br />DELETE FROM `tickets` WHERE `id` = 1007729<br />

Actually this all so far. Lesson learned for me: A mysql binlog might get you into serious trouble when firing a MySQL server with INSERT and DELETE on the same rows as the linear binlog file can fail the correct statement order, which might be a result of different MySQL threads and an unclean log behavior. I have not yet found a generic solution for the problem but I am looking forward to it.

Ubuntu Karmic Koala and Ubuntu Berlin enter metro system again

Once again, „Berliner Fenster„, the company behind the Berlin inside metro tv advertising system was so kind to provide the Ubuntu Berlin Karmic Koala release party at c-base and the release of Ubuntu with it’s own spot. It  will be shown for three days from 29th to 31th this month every 15 minutes on hundreds of screens viewable by more than a million passengers:

If you cannot see the embedded spot, click on this link.

It’s really nice to have this great audience-appealing support once again, as it presents Ubuntu in a totally non-techie medium, reaching an audience that is hard to reach and that has time to consume  input – as the screens inside the metro trains are quite appealing.

I’ll see if I can post some real life pictures of the spot soon.

sync ruby gems between different installed ruby versions

If you are in the Ruby business (which probably means „in the Ruby on Rails business“ nowadays) sooner or later you’ll have to play around with different Ruby versions on the same machine as you might run into crashing ruby processes or performance issues . At least you’ll notice that running the standard Debian/Ubuntu Ruby versions might get you into serious trouble as it is several times slower than a manually compiled version (for reference see this launchpad bug and this blog entry.).

So a common situation with is: you have Ruby and a lot of Ruby gems installed and need to switch to a different Ruby version while making sure that you have all gems installed in the new version that you had in the old version. As gems differ from version to version you should also be interested in installing exactly the same gem versions again and not justing doing a install of all recent versions.

As far as I know there is no official way of syncing gems between two ruby installations. So the common way is something like asking ruby for a list of currently installed gems like

$ gem list
*** LOCAL GEMS ***
actionmailer (2.3.2, 2.2.2, 2.1.1)
actionpack (2.3.2, 2.2.2, 2.1.1)
activerecord (2.3.2, 2.2.2, 2.1.1)
activeresource (2.3.2, 2.2.2, 2.1.1)
activesupport (2.3.2, 2.2.2, 2.1.1)
ZenTest (4.0.0)

and then running a

$ gem install actionmailer -v 2.3.2
$ gem install actionmailer -v 2.2.2
$ gem install actionmailer -v 2.1.1
$ gem install Zentest -v 4.0.0

for every gem and every gem version you probably need. As a couple of gems are native extensions they’ll get compiled and you need to wait some seconds or minutes.

As I had to do this task more than once I wrote a small wrapper script that automates the process completely by fetching the list of gems and installing them again on another ruby version:

${GEM_FROM} list | sed -n '4,$ p' | \
 while read gem versions; do
  for version in $(echo ${versions} | sed "s/[(),]//g"); do
  echo ${gem} ${version}
  ${GEM_TO} install --no-rdoc --no-ri ${gem} -v ${version}

The script uses some regular expression sed magic, friendly tweaked by Mnemonikk (thank you). Please note, that I prefer not to install rdoc and ri, as it saves time and disk space. Feel free to change this to your needs.

The only caveeat in this script are gems that cannot be installed as they come from unknown external repositories or were manually downloaded/installed. Therefore try to make sure to check this after a run of the gem sync script – it won’t stop when a gem cannot be installed which is intended behaviour.

So far about this. Hope, it helps you out when dealing with different Ruby versions. Do you have similar best practices for keeping Ruby gems in sync?

A quick note on MySQL troubleshooting and MySQL replication

PLEASE NOTE: I am currently reviewing and extending this document.

While caring for a remarkable amount of MySQL server instances, troubleshooting becomes a common task. It might of interest for you which

Recovering a crashed MySQL server

After a server crash (meaning the system itself or just the MySQL daemon) corrupted table files are quite common. You’ll see this when checking the /var/log/syslog, as the MySQL daemon checks tables during its startup.

Apr 17 13:54:44 live1 mysqld[2613]: 090417 13:54:44 [ERROR]
  /usr/sbin/mysqld: Table './database1/table1' is marked as
  crashed and should be repaired

The MySQL daemon just told you that it found a broken MyISAM table. Now it’s up to you fixing it. You might already know, that there is the „REPAIR“ statement. So a lot of people enter their PhpMyAdmin afterwards, select database and table(s) and run the REPAIR statements. The problem with this is that in most cases your system is already in production – for instance is up again and the MySQL server already serves a bunch of requests. Therefore a REPAIR request gets slowed down dramatically. Consider taking your website down for the REPAIR – it will be faster and it’s definitely smarter not to deliver web pages based on corrupted tables.

The other disadvantage of the above method is, that you probably just shut down your web server and your PhpMyAdmin is down either or you have dozens of databases and tables and therefore it’s just a hard task to cycle through them. The better choice is the command line in this case.

If you only have a small number of corrupted tables, you can use the „mysql“ client utility doing something like:

$ mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 10
Server version: 5.0.75-0ubuntu10 (Ubuntu)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> REPAIR TABLE database1.table1;
| Table              | Op     | Msg_type | Msg_text |
| database1.table1   | repair | status   | OK       |
1 row in set (2.10 sec)

This works, but there is a better way: First, using OPTIMIZE in combination with REPAIR is suggested and there is a command line tool only for REPAIR jobs. Consider this call:

$ mysqlcheck -u root -p --auto-repair --check --optimize database1
Enter password:
database1.table1      OK
database1.table2      Table is already up to date

As you see, MySQL just checked the whole database and tried to repair and optimize it.

The great deal about using „mysqlcheck“ is, that it can also be run against all databases in one run without the need of getting a list of them in advance:

$ mysqlcheck -u root -p --auto-repair --check --optimize \

Of course you need to consider if an optimize of all your databases and tables might just take too long if you have huge tables. On the other hand a complete run prevents of thinking about a probably missed table.


nobse pointed out in the comments, that it’s worth having a look at the automatic MyIsam repair options in MySQL. So have a look at them if you want to automate recovery:


Recovering a broken replication

MySQL replication is an easy method of load balancing database queries to multiple servers or just continuously backing up data. Though it is not hard to setup, troubleshooting it might be a hard task. A common reason for a broken replication is a server crash – the replication partner notices that there are broken queries – or even worse: the MySQL slave just guesses there is an error though there is none. I just ran into the latter one as a developer executed a „DROP VIEW“ on a non-existing VIEW on the master. The master justs returns an error and ignores. But as this query got replicated to the MySQL SLAVE, the slave thinks it cannot enroll a query and immediately stopped replication. This is just an example of a possible error (and a hint on using „IF EXISTS“ as often as possible).

Actually all you want to do now, is telling the slave to ignore just one query. All you need to do for this is stopping the slave, telling it to skip one query and starting the slave again:

$ mysql -u root -p
mysql> STOP SLAVE;

That’s all about this.

Recreating databases and tables the right way

In the next topic you’ll recreate databases. A common mistake when dropping and recreating tables and databases is forgetting about all the settings it had – especially charsets which can run you into trouble later on („Why do all these umlauts show up scrambled?“). The best way of recreating tables and databases or creating them on other systems therefore is using the „SHOW CREATE“ statement. You can use „SHOW CREATE DATABASE database1“ or „SHOW CREATE TABLE database1.table1“ providing you with a CREATE statement with all current settings applied.

mysql> show create database database1;
| Database  | Create Database                                                    |
| database1 | CREATE DATABASE `database1` /*!40100 DEFAULT CHARACTER SET utf8 */ |
1 row in set (0.00 sec)

The important part in this case is the „comment“ after the actual create statement. It is executed only on compatible MySQL server versions and makes sure, your are running utf8 on the database.

Keep this in mind and it might save you a lot of trouble.

Fixing replication when master binlog is broken

When your MySQL master crashes there is a slight chance that your master binlog gets corrupted. This means that the slaves won’t receive updates anymore stating:

[ERROR] Slave: Could not parse relay log event entry. The possible reasons are: the master’s binary log is corrupted (you can check this by running ‚mysqlbinlog‘ on the binary log), the slave’s relay log is corrupted (you can check this by running ‚mysqlbinlog‘ on the relay log), a network problem, or a bug in the master’s or slave’s MySQL code. If you want to check the master’s binary log or slave’s relay log, you will be able to know their names by issuing ‚SHOW SLAVE STATUS‘ on this slave. Error_code: 0

You might have luck when only the slave’s relay log is corrupted as you can fix this with the steps mentioned above. But a corrupted binlog on the master might not be fixable though the databases itself can be fixed. Depending on your time you try to use the „SQL_SLAVE_SKIP_COUNTER“ from above but actually the only way is to setup

Setting up replication from scratch

There are circumstances forcing you to start replication from scratch. For instance you have a server going live for the first time and actually all those test imports don’t need to be replicated to the slave anymore as this might last hours. My quick note for this (consider backing up your master database before!)

slave: STOP SLAVE;
slave: SHOW CREATE DATABASE datenbank;
slave: DROP DATABASE datenbank;
slave: CREATE DATABASE datenbank;

master: DROP DATABASE datenbak;
master: SHOW CREATE DATABASE datenbank;
master: CREATE DATABASE datenbank;

slave: CHANGE MASTER TO MASTER_USER="slave-user", \
MASTER_PASSWORD="slave-password", MASTER_HOST="";

You just started replication from scratch, check „SHOW SLAVE STATUS“ on the slave and „SHOW MASTER STATUS“ on the master.

Deleting unneeded binlog files

Replication needs binlog files – a mysql file format for storing database changes in a binary format. Sometimes it is hard to decide how many of the binlog files you want to keep on the server possibly getting you into disk space trouble. Therefore deleting binlog files that have already been transferred to the client might be a smart idea when running low on space.

First you need to know which binlog files the slave already fetched. You can do this by having a look on „SHOW SLAVE STATUS;“ on the slave. Now log into the MySQL master and run something like:

mysql> PURGE BINARY LOGS TO 'mysql-bin.010';

You can even do this on a date format level:

mysql> PURGE BINARY LOGS BEFORE '2008-04-02 22:46:26';


The above hints might save you same time when recovering or troubleshooting a MySQL server. Please note, that these are hints and you have – at any time – make sure, that your data has an up to date backup. Nothing will help you more.

Ubuntu developers visiting Ubuntu Berlin and c-base – plus interview with Mark Shuttleworth

A couple of months ago I started annoying people by telling them, I’d like to show the Ubuntu Berlin community and c-base to Mark Shuttleworth as he is interested in community personally on one side and Ubuntu Berlin is a great example on the other. So this’s rather inviting an important member of the community than celebrating a „meet and greet“. Of course telling people plans like this makes them smile, but when you raise your finger and say „It will happen“ with certainty, they’ll get uncertain. So the plan was actually to invite Mark to one of our great traditional release parties which you shouldn’t miss when you are around Berlin at release time.

By chance the Ubuntu Developer Sprint happened to be in Berlin for the next Jaunty release this week. If I got it right, the Canonical Ubuntu developers meet five days around two weeks before a release feature freeze and work in groups and issues that need to be decided/designed or just fixed immediatly. The incredible Daniel Holbach had the idea of inviting the bunch of developers right into the c-base after their work. So he did and we scheduled it for an evening when the Ubuntu Berlin crew also meets at c-base for their monthly jour fix.

We did not announce this meeting externally as we tried to make the whole evening as comfortable as possible for everyone. And we did, I think. Right in time at 19:30 this Wednesday evening about thirty Ubuntu developers entered the c-base. Just among them Mark who seemed to like the whole c-base hackerspace, Ubuntu Berlin, community, space and future thing a lot. The Canonical crowd got several guided tours through the whole base by __t, while housetier provided the others with „German beer“ and Club Mate. We had a lot of chats in smaller groups, things, you always wanted to ask the developer of your choice and just relaxed smalltalks about space, canooeing the c-base project „OpenMoon“ (trying to send a rocket to the moon), and more. Mark seemed to be amazed about asking people why they joined to Ubuntu Berlin team, what they were currently doing and so on – so, what the community thing is about right here and right now.

We had no schedule for the evening. Therefore we spent about two and a half hour at c-base without any official part and a cosy diner in smaller groups afterwards. We only asked some people for an interview for the c-base statement studio channel where already people like Nokias Peter Schneider and Mozillas CEO John Lilly showed up. Mark took the time for an interview by jocognito. The results of this short talks are already online on Youtube:

Interview #1: (when the inbound video doesn’t show up, click here)

Interview #2: (when the inbound video doesn’t show up, click here)

Another interview with Jorge Castro and James Westby has also been taped and will be published soon. Funny guys talking about Ubuntu on the moon. I’ll post the Youtube links, when the edited version is online.
So what’s next? I hope everybody liked (Ubuntu) Berlin and c-base and we got a good start for possible events in the future.
Thanks again to Daniel who initiated the visit!

Ubuntu Berlin raising money for a video projector – want to support?


I already mentioned that one goal of Ubuntu Berlin for this year is raising enough money for a new video projector. The „Ubuntu Berlin“ team is quite active with its release parties, bug jams, lectures and workshops and most of these events require us to present software to an audience (surprise!). We are really happy about having the possibility to use the infrastructure of the well known c-base, but the video projector there is really close to its end of life-cycle and more or less unusable (too dark, unsharp, etc.).

Therefore we decided to buy a new video projector funded by the community. The „ubuntu Deutschland e.V.„, the main German LoCo team, if you can say so, and legal association behind major parts of the German community, is able to collect the money and can even issue an official contribution receipt as this is tax deductable in Germany.

So if you have some money left and want to support one of the most active teams in the Ubuntu community, feel free to donate some money to:

account holder: ubuntu Deutschland e.V.
bank: HVB Nürnberg
account number: 382916859
bank code: 76020070
IBAN: DE44760200700382916859

Swift (BIC)
purpose of transfer: „Ubuntu Berlin“ (important!)

Of course you can also donate money from another country than Germany, just use the IBAN/Swift :)

For your information:

We intend to buy a projector for about EUR 500 to 700, depending on the amount of money we collect. The beamer will be a permanent loan to the c-base, located in their workshop room – therefore used for hundreds of lectures and workshops around free software. If we collect more than we actually need for the projector, we’ll spend the remaining money for other expenses connected to Ubuntu Berlin (e.g. further reconstruction of the workshop room at c-base, funding of our bigger events like LinuxTag BBQ, release parties, etc.).

Thanks for reading, and: Donating makes you really sexy :)


You can also donate money over the wire: Just send the amount of money you like to spend via PayPal to: [email protected]

Ubuntu Berlin review of 2008

End of 2008 is close and it’s time to review the year, isn’t it? As some of you might know, I am quite active in the „Ubuntu Berlin“ team, a local Berlin(Germany)-based user group, closely connected to the internationally known hackerspace c-base.

Le me just sum up what we’ve done this year:

  • about one dozen regular meetings (earch 1st Wednesday of the month) with an average of 10 to 30 visitors
  • we participated in the Ubuntu Global Bug Jam in August
  • we had about one dozen Bug Jams with Ubuntu developer Danien Holbach
  • we hosted two Ubuntu release parties (for Hardy and Intrepid) with about 200 visitors each of them and a video feature in the Berlin metro system viewable by approx 1.5 million people
  • we had several workshops and lectures for topics like „introduction to Latex“, „video editing with Cinelerra“, „fun on the command line“, „graphics and design with free software“
  • we supported the „Linuxtag 2008“ and hosted the end-of-Linuxtag BBQ with dozens of free software developpers and Ubuntu affilliated people
  • we started a „terminal project“, trying to set up Ubuntu powered touch screen terminals
  • we organised an Origami workshop as Christmas social event (ever build a Tux or star?)
  • we started a new series of events with a kind of lightning talks giving the possibility to speak just a couple of minutes about a piece of software – with really great success

So, looking back, 2008 was a successfull year for Ubuntu Berlin. We managed not to split into small groups and were able to increase the amount of members. Currently 120 people are reading our main mailing list and about 110 have joined our launchpad group. And no, they are not all nominal members.

So what are our goals for 2009?

Our main goal will be focussing on Ubuntu related topics. It’s hard to have clearly structured, topic related discussions on a mailing list with more than 100 readers. We already split our list into three lists and try to calm down every „religious“ debate.

We will definitely host two major release parties again while trying to make them even better. There is space for improvements in the lecture schedule and quality, though we were really fine with the last two parties. I guess we will also organise something like a BBQ for Linuxtag 2009, again.

I hope we’ll manage to provide at least one workshop per month like we did in the last months as this really brings people to the machine, using Ubuntu and free software. The last workshops took round about 90 minutes. We are also trying to host longer workshops (approx four ours) giving you real hands on experience. Moreover we’ll continue with the lightning talk sessions as this is a real neat type of event.

Together with Daniel Holbach we’ll continue with the Ubuntu Berlin Bug Jams while trying to shift them towards a real practical experience solving dozens of bugs each session, motivation users to join the Bug Triage team.

And we’ll continue to collect donations as we urgently need things like a new video beamer for our talks and similar events. We hope to find a sponsor providing Ubuntu Berlin with a monthly budget.

So, let’s start into 2009 and make it an Ubuntu year, again. It’s up to us convincing people of using Ubuntu, improving Ubuntu and spreading the word again.

Ubuntu Intrepid Ibex Release Party on 1st of November at c-base (Berlin)

So, half a year has passed and it’s time again to celebrate a new Ubuntu release. This is an invitation for you, your friends and any other human being around to join our „Ubuntu Intrepid Ibex Release Party“ on Saturday, the 1st of November, at the sunken starship c-base (Rungestraße 20), starting at 4pm.

Again, a couple of lectures will be held – ranging from new features in Intrepid (that’s my part, I’ll give the lecture at the same day also at the BLIT), over Gnome eyecandy stuff to presentations of Freifunk project and the DeepaMehta semantic desktop (don’t miss!) directly from it’s lead developer. So either if you are new to Ubuntu and want to make first steps and contact other Ubuntu users or you are a Linux guru – you’ll find somebody to have a chat with, I promise. There’ll even be a „tux tinker corner“ where you or your girl friend have the possibility to try out some Tux Origami.

The event’ll mainly be in German, but a lot of people are speaking English, so don’t hesitate asking for help/translations.

Entrance is free, there is a free wifi, so feel free to bring your notebook in. You can also „buy“ a freshly burned Ubuntu/Kubuntu/Xubuntu cd for a service charge of 1 Euro or check the Ubuntu merchandising table.

See you there?

Official Party Announcement Page
how to get to c-base?
Ubuntu Berlin

Your mom runs Ubuntu? Join the team? She doesn’t yet? Help her. And join the Team!

After having a lot of chats with friends I’s surprised how many of them said „Yes, I installed Ubuntu on my mom’s pc.“ So I assume there is a growing user group which will never pop up in blogs, user groups or fairs: your mom, the mom of your friend and mine. As this is a user group bringing Ubuntu to the real end user, I think it’d be nice showing that these users exist and motivating other Ubuntu users to let their mothers join the Ubuntu crowd just by installing them Ubuntu, wouldn’t it?

So join the just created launchpad team „my mom runs ubuntu“ by clicking here and show the public that you made another mom happy. I promise I’ll change the launchpad group branding as soon as possible.

My mom runs Ubuntu. And yours?