When backups fail: A mysql binlog race condition

March 2nd, 2010

Today I ran into my first MySQL binlog race condition: The initial problem was quite simple: A typical MySQL master->slave setup with heavy load on the master and nearly no load on the slave, which only serves as a hot fallback and job machine, showed differences on the same table on both machines. The differences showed up from time to time: entries that have been deleted from the master were still on the slave.

After several investigations I started examining the MySQL binlog from the master – a file containing all queries that will be transferred to the slave (and executed there if they don’t match any ignore-db-pattern). I grepped for ids of rows that have not been deleted on the slave as I’s interested if the DELETE statement was in the binlog. In order to read a binlog file just use “mysqlbinlog” and parse the output with grep, less or similar. To my surprise I found the following entries:

$ mysqlbinlog mysql-complete-bin.000335 | grep 1006974
DELETE FROM `tickets` WHERE `id` = 1006974
SET INSERT_ID=1006974/*!*/;

As “SET INSERT_ID” is a result of an INSERT statement it was clear, that MySQL wrote the INSERT => DELETE statements in the wrong order. As INSERT/DELETE sometimes occur quite fast after each other and several MySQL  threads are open in the same MySQL server, you might run into a rare INSERT/DELETE race condition as the master successfully executes them, while the slave receives them in the wrong order.

As a comparision this is a normal order of INSERT and DELETE (please note that the actual INSERT is not displayed here):

$ mysqlbinlog mysql-complete-bin.000336 | grep 1007729<br />SET INSERT_ID=1007729/*!*/;<br />DELETE FROM `tickets` WHERE `id` = 1007729<br />

Actually this all so far. Lesson learned for me: A mysql binlog might get you into serious trouble when firing a MySQL server with INSERT and DELETE on the same rows as the linear binlog file can fail the correct statement order, which might be a result of different MySQL threads and an unclean log behavior. I have not yet found a generic solution for the problem but I am looking forward to it.

“human” Ubuntu Karmic Koala release notes – a reminder

October 29th, 2009

Hi,

Karmic is to be released any minute now, so just as a reminder:

Over the last weeks I started writing “human” release notes that concentrate on the most important changes that impact the users work. They are available on the Ubuntu Wiki:

https://wiki.ubuntu.com/KarmicKoala/HumanReleaseNotes

If you like, you can add a screenshot of Empathy (and Pidgin?) video chat in action or send them to me, so I’ll upload them.

Please note, that Ernst Sjöstrand just dropped me a line, that he had the same idea of writing “human” release notes, but focussed on main packages and there forked the work on

https://wiki.ubuntu.com/KarmicKoala/HumanReleaseNotesv2

In my eyes the human release note project is yet at an early stage but I got a lot of positive feedback and think the work can be continued with the next Ubuntu Lucid Lynx release in an advanved way with more contributors.

Best,

ccm.

Ubuntu Karmic Koala and Ubuntu Berlin enter metro system again

October 26th, 2009

Once again, “Berliner Fenster“, the company behind the Berlin inside metro tv advertising system was so kind to provide the Ubuntu Berlin Karmic Koala release party at c-base and the release of Ubuntu with it’s own spot. It  will be shown for three days from 29th to 31th this month every 15 minutes on hundreds of screens viewable by more than a million passengers:

If you cannot see the embedded spot, click on this link.

It’s really nice to have this great audience-appealing support once again, as it presents Ubuntu in a totally non-techie medium, reaching an audience that is hard to reach and that has time to consume  input – as the screens inside the metro trains are quite appealing.

I’ll see if I can post some real life pictures of the spot soon.

Towards a “human” Ubuntu (Karmic Koala) release note

October 1st, 2009

These days I am busy preparing my talk about news in Ubuntu Karmic Koala which I’ll present at least three times: at Ubucon (Göttingen), Ubuntu Karmic Koala Release Party (Berlin) and Brandenburger Linuxtag BLIT (Potsdam). This is the third time I am going to present news in an Ubuntu release to a broader audience, therefore I gained experiences in researching information about an upcoming or just issued Ubuntu release.

While collecting new features and changes in Ubuntu I continue to notice that I am quite unsatisfied with the information provided for a release. Let me tell you why and what I suggest to change this.

The Status Quo

Ubuntu is released twice a year. Every release means thousands of changed packages, fixed bugs, enhancements and new functions. With every release a release note is added to the release overview page. Lets have a look at the first paragraphs of the most recent release note for Ubuntu Jaunty Jackalope:

These release notes document known issues with Ubuntu 9.04 and its variants.
System Requirements

The minimum memory requirement for Ubuntu 9.04 is 256 MB of memory. Note that some of your system’s memory may be unavailable due to being used by the graphics card. If your computer has only the minimum amount of memory, the installation process will take longer than normal, but will complete successfully, and the system will perform adequately once installed.

Systems with less memory may be able to select “Install Ubuntu” from the boot menu to run just the installer, rather than the whole desktop, or may be able to use the alternate install CD.

Installation
Recommended packages installed by default

In accordance with the Debian Policy Manual (which says “The ‘Recommends’ field should list packages that would be found together with this one in all but unusual installations”), the package management system now installs packages listed in the Recommends: field of other installed packages as well as Depends: by default. If you want to avoid this for specific packages, use apt-get –no-install-recommends; if you want to make this permanent, set APT::Install-Recommends “false”; in /etc/apt/apt.conf. Be aware that this may result in missing features in some programs.

Interesting and useful information, in a way. But in my eyes not what I am looking for when looking for a “human” release note. The current release notes are focussed on technical details and written in a language that might appeal an advanced audience like linux pros, developpers and system administrators. These are important target groups, of course, but they are not the only target group for a release, are they? Actually I hope, these people, and I count myself to them, are only a minority of Ubuntu users.

Who do we want to talk to?

So what is a suitable and broad target group for a release (note) that does not focus on techies? The answer is fairly easy: It’s the main group of average users who just wants to do things with their machines: office work, surfing the web, using social services, organizing their family pictures, watching video streams and so on. They don’t care that much about version numbers, framework changes and new apis. They use the computer as a tool and that’s exactly what makes Ubuntu “human” for me and for them.

If these users consider upgrading to or installing a new release, where do they get hints about features that might convince them of doing so? Well, from online and print magazines as well as blogs featuring the new release. But as these texts also need to be written – where do the journalists and bloggers get their background from? Well, they need to collect small pieces of information on the web, which is an exhausting job as there is no official and comprehensive document and you need to collect and prove every bit of information while still risking missing important facts. So the second group of people a release note might address besides average users are people writing texts: journalists and blogger.

Let’s call these groups non-technical, though they might be pros. They play a non-technical here role as even if they are technical journalists or developers they want to know about features that support their day to day work.

What do we want to tell?

So let’s rethink a release note from user’s point of view. Let’s say the user uses OpenOffice.org for writing documents and doing spreadsheet calculation, Firefox for surfing, Evolution for Mail, Pidgin for Chat. What might be facts that’d get his attention in the upcoming Karmic release? Let’s try a too short example list of interesting facts about the upcoming Karmic Koala release (PLEASE NOTE: These are just sample notes – neither long enough nor well proved enough, so don’t complain about them, please ;) )

  • The new “Ubuntu Software Center” has been introduced as a replacement for the “Add Software” Synaptic (?) stuff and comes with a task oriented clean interface and even showing screenshots of a lot applications before installing them. Software installation has never been so easy.
  • The network manager has been revamped. Mostly unnecessary details have been removed from the standard dialogues. Using the network manager got more intuitive. Dealing with wifi networks is easier to manager as you have to make less decisions.
  • The computer janitor, a tool for removing outdated and unused software, got extended so the interface displays more useful information making it easier to decide what to remove.
  • [...]
  • Firefox 3.5 is the new default browser. Besides increased stabililty and faster page rendering it brings a private surf mode, leaving no traces on disk, you can use specific searches for bookmarks and history in the address bar, [...]
  • Evolution is now able to import data from the hamster applet via iCal.
  • The default chat client Pidgin has been replaced by Empathy, which is a port of the Gnome Desktop environment and well integrated. Empathy comes with a clean interface, video and audio chat features. You should give it a try. Pidgin is still available via the Software Center of course.
  • [...]

I think you get what I am trying to tell: Giving the user facts about features that have impact on his user experience and his daily work. There is even an example of a quite felicitous note release that is closely connected to an Ubuntu release: The Gnome Project presents outstanding multi-language (!) release notes with appealing language, screenshots and a clean structure. Have a look at the release note for Gnome 2.28 that gets shipped with Ubuntu 9.10.

Ubuntu cannot copy these notes at a whole as they some parts differ in Ubuntu. For instance Empathy has no geolocation bindings so far in Karmic (think that has been ripped as it would exceed the size for the standard iso image). And Ubuntu is not only Gnome, like Kubuntu is not only KDE.

Towards a collective, human release note

Putting my notes together, here is my pragmatic-let’s-make-improve-it-suggestion:

The Ubuntu project already makes massive usage of wikis. As a lot of people collect useful and user-oriented information about a release and users as well as journalists and bloggers are happy about structured collections of features, we should combine our efforts in documenting what’s so special about a release. The following four types of features might be of broader interest:

  1. Daily use applications like all applications that ship with the standard iso as well as often installed ones like Thunderbird, VLC media player, and so on.
  2. Changes within Gnome that made their way into Ubuntu like new applets, improved tools and settings.
  3. Important changes on the technical framework having an impact on user experience (like better boot times due to upstart, faster file access by using ext4, …)
  4. New bleeding edge features that show the upcoming feature but are not visible by default (like gnome-shell).

In fact a document containing all those information can get quite long. But hey: It would finally be a decent starting point, gives user to opportunity to read on and bloggers and journalists a kick start right from the scratch.

Doing things right here right now

After writing all this I’d like to channel interests into an pragmatic approach. I already emphasized how useful a wiki’d be in our case of collaboratively writing a release note for the masses. So let’s start right away, now, and hit the “edit” button on:  The Karmic Koala “Human” Release Notes. I just added some topics and text snippets as a starting point. I think the structure can work this way. Let me know, what you think. At the end it would be great having interesting screen shots showing some of the news and changes.

As I’ll attend the Ubuntu Global Jam this Saturday, I’ll use the time for extending the document. A perfect day for this task.

Keep in mind who we want to reach

Keeping an eye on the target groups – users, journalists and bloggers – helps keeping the “human” release note straight forward. Friendly language is fine here, talking about frameworks not – at this place. Don’t forget: We still have a technical release note.

Now it’s up to you: What do you think about my proposal? Would you like to join working on a more user focussed document? Let me know.

[update]

Wouter sent a hint on the http://www.ubuntu.com/products/whatisubuntu/904features/ page. Indeed I did not know it so far (I did not find an easy click-way from the Ubuntu home page to this document so far), and am investigating the document right now. So far I can tell that it is, of course, more suitable to my requirements than any other document so far, but I already guess that it is still not exactly what I need. Be it the late publishing of the document regarding the release date.

[update2]

After having reviewed the document I still miss a “human” release note. Why? Well, the document from above is rather a description of a current release, repeating a lot of already known facts from previous releases and therefore doesn’t focus on changes in the release. Nonetheless I totally agree to Wouter, this could be more prominent, but we still another type of release notes.

Global Jam Participation – Ubuntu Berlin is prepared – are you?

September 22nd, 2009

(Wow, I needed to type “Global Jam” more than once as I’s used to write “Global Bug Jam”.)

I am happy looking forward to the third “Ubuntu Global Jam” taking place from 2nd to 4th of October 2009. “We”, meaning “Ubuntu Berlin“, Daniel Holbach, Benjamin Drung and me, already managed to prepare the local Jam session as far as possible:

  1. We found a place with power supply and network uplink: the c-base, where Ubuntu Berlin became some kind of resident.
  2. We decided to schedule our Global Jam for Saturday, the 3rd of October, from 11 am to 7 pm, which should allow people with different personal schedules to join us at least for some hours.
  3. We added our Global Jam to the official team list.
  4. We set up a core team with responsibilities for different aspects of the Jam. Daniel Holbach and Benjamin Drung will care for bug triaging, bug squatting and packaging while I’ll work on and present translation and documentation.
  5. We announced our Global Jam participation and invited more people to join us on several mailing lists (from Ubuntu Berlin lists over local Berlin Linux lists to c-base lists).
  6. We started to announce our invitation on different twitter feeds, our facebook group, identi.ca and some blogs. One of them you are currently reading :)
  7. Last but not least I dropped some lines on the Ikhaya-suggest-an-article box, one of the most read German Ubuntu ressources.
  8. We will continue inviting people on personal basis as there are a bunch of people able to support us and just need a small friendly push… :)

I admit I am curious about this years participation and impact. And you? What are your hints for setting up a Global Jam session? Or do you need any more advices? Let me know. I am curious. Really.

usability as blocker? of course. – UbuntuOne and the bandwith limit

August 18th, 2009

Recently I ran into a missing feature of the UbuntuOne client. If you don’t know yet: UbuntuOne is the Canonical driven online space for storing and saving data. It’s a commercial service with a free entry level like similar offers from DropBox. The UbuntuOne client is a small open source application that cares for the synchronisation of files. So when you put a file in a specific folder, the client pushes it to the server or pulls changes in the same way.

The missing feature was a small one: The ability to limit the upload speed. As you might know, uploading large files can nearly block you internet access as your machine isn’t able to send back packages in time. So limiting upload speed either automatically or with a specific setting helps you sending data to the net while still being able doing other things.

While the fact that the feature is somehow is maybe rather not of interest for you, the interesting part is to see how the feature request evolves in the Ubuntu bug tracker. I filed a request myself (#375328) and ran into a – technically correct – discussion about the necessity of implementing network speed limits inside applications. Of course you have the ability to use the Linux kernels traffic shaping features or even a more centralized setup in your local network. While these arguments are absolutely right from an administrator’s point of view they are nearly incorrect from an end user’s side. And end user shouldn’t really care about lan traffic shaping setups or need to know about the Linux kernel’s traffic shaping features.

So while more beta testers filed similar feature requests and (#381348) got the main ticket, the importance of this issue remained in the discussion. I’s surprised to see that the request got recently tagged as “karmic-blocker” meaning it has to be done before Ubuntu Karmic Koala is released. While the tag was removed temporarily as a reason was missing, Elliot Murphy filed it later, stating

“if we don’t have bandwidth limiting and the user fires up their laptop on a slow connection (maybe an edge connection via their mobile phone), and the syncdaemon will use all available bandwidth and cripple any other applications that need some bandwidth. we’ve gotten bug reports from users already complaining about this being so bad that DNS requests are taking forever to get through. so, I think the syncdaemon having some semi-intelligent bandwidth limiting (like the suggestion of monitoring the transmit queue depth) is a karmic blocker.”
source: https://bugs.launchpad.net/ubuntuone-client/+bug/381348

The point about all this is: The nearly tiny feature request of adding a bandwith control to a small client got a blocker for Karmic as it might break user experience and could lead to a lot of bug reports about slow networking connections that are rather about UbuntuOne client consuming upload speed completely. I guess the decision to handle this as a blocker might be surprising on the one side but it is a wise decision as it focusses on the users point of view on the other and finally that’s what it’s all about: the user. Isn’t it?

sync ruby gems between different installed ruby versions

June 7th, 2009

If you are in the Ruby business (which probably means “in the Ruby on Rails business” nowadays) sooner or later you’ll have to play around with different Ruby versions on the same machine as you might run into crashing ruby processes or performance issues . At least you’ll notice that running the standard Debian/Ubuntu Ruby versions might get you into serious trouble as it is several times slower than a manually compiled version (for reference see this launchpad bug and this blog entry.).

So a common situation with is: you have Ruby and a lot of Ruby gems installed and need to switch to a different Ruby version while making sure that you have all gems installed in the new version that you had in the old version. As gems differ from version to version you should also be interested in installing exactly the same gem versions again and not justing doing a install of all recent versions.

As far as I know there is no official way of syncing gems between two ruby installations. So the common way is something like asking ruby for a list of currently installed gems like

$ gem list

*** LOCAL GEMS ***

actionmailer (2.3.2, 2.2.2, 2.1.1)
actionpack (2.3.2, 2.2.2, 2.1.1)
activerecord (2.3.2, 2.2.2, 2.1.1)
activeresource (2.3.2, 2.2.2, 2.1.1)
activesupport (2.3.2, 2.2.2, 2.1.1)
[...]
ZenTest (4.0.0)

and then running a

$ gem install actionmailer -v 2.3.2
$ gem install actionmailer -v 2.2.2
$ gem install actionmailer -v 2.1.1
[...]
$ gem install Zentest -v 4.0.0

for every gem and every gem version you probably need. As a couple of gems are native extensions they’ll get compiled and you need to wait some seconds or minutes.

As I had to do this task more than once I wrote a small wrapper script that automates the process completely by fetching the list of gems and installing them again on another ruby version:

#!/bin/sh
GEM_FROM=/path/to/old/gem
GEM_TO=/path/to/new/gem
${GEM_FROM} list | sed -n '4,$ p' | \
 while read gem versions; do
  for version in $(echo ${versions} | sed "s/[(),]//g"); do
  echo ${gem} ${version}
  ${GEM_TO} install --no-rdoc --no-ri ${gem} -v ${version}
 done
done

The script uses some regular expression sed magic, friendly tweaked by Mnemonikk (thank you). Please note, that I prefer not to install rdoc and ri, as it saves time and disk space. Feel free to change this to your needs.

The only caveeat in this script are gems that cannot be installed as they come from unknown external repositories or were manually downloaded/installed. Therefore try to make sure to check this after a run of the gem sync script – it won’t stop when a gem cannot be installed which is intended behaviour.

So far about this. Hope, it helps you out when dealing with different Ruby versions. Do you have similar best practices for keeping Ruby gems in sync?

Ubuntu Jaunty Jackalope on Berlin metro system

April 23rd, 2009

Again I am happy to announce, that Berliner Fenster, the company behind the Berlin metro tv advertisement system, viewed by approx. 1.5 million people a day, was so kind of providing Ubuntu and Ubuntu Berlin with a spot for the release of Ubuntu Jaunty Jackalope and the release party hosted by Ubuntu Berlin at c-base:

If you cannot see the embedded spot, click on this link.

The spot runs for three days, showing Ubuntu to a remarkable amount of passengers.

Please note: The background image is from Marvin Kubiak, you can find it among other interesting Jaunty background images on:

https://wiki.ubuntu.com/Artwork/Incoming/Jaunty/AlphaBackgrounds

Happy release!

What do director Tom Tykwer and Ubuntu have in common?

April 22nd, 2009

What do director Tom Tykwer (”The International“) and Ubuntu have in common? Well, they seem to share the same passion for commitment in developing countries by focusing on media. This similarity came up when I read the schedule for the one day conference “Jour Fixe – media and development“.

While the first talk, held by Andrea Goetzke und Geraldine de Bastion from newthinking communications, titled “Ubuntu and the free toaster“, deals with free software and digital culture in Africa, the second talk, held by director Tom Tykwer and titled “The Making of Soulboy. A movie production in the slums of Nairobi“, deals with a movie project in Nairobi (and Tom Tykwers engagement in artistic education for youngsters in Africa with the ngo “one fine day”).

The “Jour Fixe” is scheduled for this Friday, the 24th of April, and takes place at “Hamburger Bahnhof” in Berlin/Germany. As I’m going to attend the conference, I’ll try to report back a short resume. Please note that there are more interesting talks besides Ubuntu/Tykwer – I just wanted to point out the interesting coincidence.

possibly scrambled qt4 apps in Ubuntu Jaunty Jackalope while using VRGB and LCD

April 21st, 2009

In less than 48 hours Ubuntu Jaunty Jackalope will be released officially! While using Jaunty for more than a month I am pleased to see this great new release getting it’s way to a huge usership. While preparing my small “new features in Ubuntu Jaunty Jackalope” talk for the “Ubuntu Berlin” release party at c-base, I’d like to point out one open bug, that will make it into the release and might get some users into strange trouble:

When you have – for whatever reason (either by chance, clicking around or due to having a rotated display) – changed the display mode of Ubuntu to VRGB or VBGR (you can easily do this in the “System>Preferences>Appearance>Fonts” menu) and additionaly turned on the lcd/subpixel mode, probably all of your qt4 applications – even qtconfig – will be totatlly scrambled and unreadable. The bug might hit you under Ubuntu (without a blue “K” in the beginning) as you probably run something like VirtualBox, Skype or even Amarok in your Gnome environment. It will probably look like this:

Screenshot showing the scrambled qt application
Screenshot showing the scrambled qt application

But: Don’t panic! This bug only occurs when you changed some defaults in the Appearance/Fonts settings to rather uncommon or often maybe useless settings and switching back is fairly easy. But as I ran into this and put some effort into the corresponding bug on launchpad Subpixel/Lcd mode with VRGB/VBGR makes qt4 applications on Jaunty unreadable, I noticed, that a remarkable bunch of users posted duplicate bug reports or commented the report.

So if you are running into this error, all you have to do, is:

  1. exit all qt applications,
  2. go into “System>Preferences>Appearance>Fonts”,
  3. change VRGB/BGR to RGB/BGR.

That’s all. If you want to examine the bug, do the following (don’t do this at home):

  1. exit all qt applications,
  2. go into “System>Preferences>Appearance>Fonts”,
  3. change RGB/BGR to VRGB/BGR,
  4. switch to lcd/subpixel mode.

Afterwards your qt applications will look scrambled. If you don’t know, which one to run for a test, you probably have “qtconfig” installed.

(I had a little chat with some developers about the severity of this bug. Though I think it’d would be definitely better not having it in a release, I understand the point, that the environment must be set up in a rather special way to run into this and it popped up quite late. Therefore it did not make it into the release critical bugs, but I am quite sure, it will be fixed soon.)

That’s all about this. I wish you a smooth upgrade to Jaunty – see you there :)

[update 2009-04-03]

As you can read on https://bugs.launchpad.net/ubuntu/+source/qt4-x11/+bug/334657, a patch has been committed to the Ubuntu repositories, that has been found here:

http://cvs.fedoraproject.org/viewvc/rpms/qt/devel/qt-x11-opensource-src-4.5.0-disable_ft_lcdfilter.patch?revision=1.1&view=markup

So the fix as an update is coming closer…