usability as blocker? of course. – UbuntuOne and the bandwith limit

Recently I ran into a missing feature of the UbuntuOne client. If you don’t know yet: UbuntuOne is the Canonical driven online space for storing and saving data. It’s a commercial service with a free entry level like similar offers from DropBox. The UbuntuOne client is a small open source application that cares for the synchronisation of files. So when you put a file in a specific folder, the client pushes it to the server or pulls changes in the same way.

The missing feature was a small one: The ability to limit the upload speed. As you might know, uploading large files can nearly block you internet access as your machine isn’t able to send back packages in time. So limiting upload speed either automatically or with a specific setting helps you sending data to the net while still being able doing other things.

While the fact that the feature is somehow is maybe rather not of interest for you, the interesting part is to see how the feature request evolves in the Ubuntu bug tracker. I filed a request myself (#375328) and ran into a – technically correct – discussion about the necessity of implementing network speed limits inside applications. Of course you have the ability to use the Linux kernels traffic shaping features or even a more centralized setup in your local network. While these arguments are absolutely right from an administrator’s point of view they are nearly incorrect from an end user’s side. And end user shouldn’t really care about lan traffic shaping setups or need to know about the Linux kernel’s traffic shaping features.

So while more beta testers filed similar feature requests and (#381348) got the main ticket, the importance of this issue remained in the discussion. I’s surprised to see that the request got recently tagged as „karmic-blocker“ meaning it has to be done before Ubuntu Karmic Koala is released. While the tag was removed temporarily as a reason was missing, Elliot Murphy filed it later, stating

„if we don’t have bandwidth limiting and the user fires up their laptop on a slow connection (maybe an edge connection via their mobile phone), and the syncdaemon will use all available bandwidth and cripple any other applications that need some bandwidth. we’ve gotten bug reports from users already complaining about this being so bad that DNS requests are taking forever to get through. so, I think the syncdaemon having some semi-intelligent bandwidth limiting (like the suggestion of monitoring the transmit queue depth) is a karmic blocker.“
source: https://bugs.launchpad.net/ubuntuone-client/+bug/381348

The point about all this is: The nearly tiny feature request of adding a bandwith control to a small client got a blocker for Karmic as it might break user experience and could lead to a lot of bug reports about slow networking connections that are rather about UbuntuOne client consuming upload speed completely. I guess the decision to handle this as a blocker might be surprising on the one side but it is a wise decision as it focusses on the users point of view on the other and finally that’s what it’s all about: the user. Isn’t it?

my package of the day – htop as an alternative top

„top“ is one of those programs, that are used quite often but actually nobody talks about. It just does its job: showing statistics about memory, cache and cpu consumption, listing processes and so on. Actually top provides you some more features like batch mode and the ability to kill processes, but it’s all quite low level – e.g. you have to type the process id (pid) of process you want to kill.

So, though an applications like top makes sense on the console, a more sophisticated one would be great, extending the basic top functionality with enhancements to it’s usage. This tool already exists: It’s the ncurses based „htop“ and we’ll have a closer look at it now.

For the beginning: Install „htop“ by running „aptitude install htop“, Synaptic or the package manager of your choice. As you can see, htop is quite colorful, which is, of course, a matter of taste. In my opinion, colors make sense, when the they mean something or provide better readability. So let’s check the output in brief:

htop1.png

At the upper left corner you see statistics about the usage of cpu cores (in my case there are two of them, marked „1“ and „2“), memory and swap statistics, while on the right side, you have the common uptime/load stats. The interesting part is the usage of colors in cpu/ram/swap bars. If you are new to htop you have to look the colors up at least once. Therefore just stroke „h“ („F1“ should work, too, but Gnome might get in your way) and you’ll see a nice explanation in the help:

htop2.png

Quite interesting is the distribution between green and red in the cpu stats, as a high kernel load often means something goes wrong (with the hardware i/o for instance). In the memory bar the real used ram is marked green – blue and orance actually could be cleared by the kernel if necessary. (People are often confused that their ram seems to be full, when calling a tool like/htop though they are not running that many programs. It’s important to understand, that the memory is also used for buffering/caching and that this memory can often be used by „real“ data later on).

So what’s the next htop feature? Use your mouse, if you like! You can test it by clicking on „Help“ on the menu bar at the bottom. Maybe while clicking around a bit you already noticed that you can also click on processes and mark them. What for? Well, htop enables you to kill processes quit easy, as you don’t have to type a process id, write a pattern or something, you just can mark them with a mouse or cursor and either click on „Kill“ in the menu or stroke the „F9“ or „k“ key. „htop“ will let you choose from a list of signals afterwards:

htop3.png

Of course you cannot kill processes that belong to your user when htop does not run as root (i.e. with „sudo“). „htop“ marks processes that belong to user it is run by with a brighter process id:

htop4.png

Sadfully this also means, that running htop as root/sudo, marks processes that belong to non-root with a darker grey. But hey, that’s a nice missing feature for patch, isn’t it?

If you like to become an advanced htop user, you can check the „Setup“ menu (click it or press the „F2“ or „S“ key). You will see a menu for configuring the output of htop, enabling you to switch off and on the display of certain information:

htop5.png

Of course you can also sort the process list (click „Sort“ or press „F6“) which give you a list of possible sort parameters:

htop6.png

In spite of this, you can switch to a process tree display and sort it by pressing one of the keys showed below:

htop7.png

So let me give you a last nice gimmick and then end for today: You can try to attach „strace“ to a running process by marking the process typing „s“. If you don’t know, what strace is, don’t bother, if you do, you will probably like this feature pretty much.

I hope you got the clue about using htop, which is a really neat, full featured console top replacement that is even worth to be used when running X as it supports mouse usage and brings everything you need while still having a small footprint. If you have alternatives, you like mention, feel free to drop them as a comment.

my package of the day: irqbalance

Maybe you are in the nice situation of running machine with multiple cpu cores having to crunch a lot of numbert or providing network daemons. Multiple cpu cores can boost your performance dramatically but there are, of course, possible issues. On is the fact, that interrupts by default are mainly called by the first cpu. As applications that aren’t able to thread correctly can stick to this cpu you might notice by a „cat /proc/interrupts“ that something goes wrong. The following is a (compressed) /proc/interrupts from a live server running for about ten weeks:

           CPU0       CPU1
  0: 1855681242        574  timer
  7:          0          0  parport0
  8:          1          0  rtc
  9:          1          0  acpi
 14:         65          0  ide0
 58:          4          0  ehci_hcd:usb1, uhci_hcd:usb2
 66:  212905125         72  3w-xxxx
 74: 1094755762          0  eth0
185:    6223686          0  uhci_hcd:usb4, eth1
193:       1978          0  uhci_hcd:usb3, libata

You can see, that actually all interrupts are called by CPU0. We want to brush this up! How? Just run a „aptitude install irqbalance“ as irqbalance promises:

Daemon to balance interrupts across multiple CPUs, which can lead to better performance and IO balance on SMP systems. This package is especially useful on systems with multi-core processors, as interrupts will typically only be serviced by the first core.

So let’s check after about one week by another „cat /proc/interrupts“:

           CPU0       CPU1
  0: 1887385089   33827155     timer
  7:          0          0     parport0
  8:          1          0     rtc
  9:          1          0     acpi
 14:         65          0     ide0
 58:          4          0     ehci_hcd:usb1, uhci_hcd:usb2
 66:  212950265   11810501     3w-xxxx
 74: 1310191290          0     eth0
169:          0          0     uhci_hcd:usb5
185:    6223686     228881     uhci_hcd:usb4, eth1
193:       1978          0     uhci_hcd:usb3, libata

Nice, isn’t it? CPU1 started to grab interrupts also. If we would reboot the server, most of the irqs would look balanced over time. (most, not all)

Please notice:

Before installing irqbalance, check your /proc/interrupts. It might be possible, that you don’t need it though you have multiple cores as there is a value „CONFIG_IRQBALANCE“ in 2.6 kernels that can be turned on.

[update]

The comments (thank you!) pointed out the following:

  • There are reports on crashed systems using irqbalance. (Though I have never seen anyone by myself)
  • Note that irqbalance is not in main – if you are using it on an important server.
  • CONFIG_IRQBALANCE seems to be enabled in Bbuntu Hardy by default.
  • There are discussions about removing CONFIG_IRQBALANCE as it is said that irqbalance is more reliable.

So it is up to you to decice which one to use!

[update2]

Actually a glance on  /boot/config-2.6.24-17-generic shows, that CONFIG_IRQBALANCE seems not to be enabled in Hardy though the balancing seems to work. Actually I am not one of the kernel guys so my investigation will take it’s time. Any hints welcome (thank you lissyx).

fast installation of apc php optimizer/cache on Debian / Ubuntu

If you want a fast installation of the php apc bytecache/optimizer for PHP5/Apache2, try the following snippet when already running a standard PHP5/Apache2 environment:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# install dependencies for compilation
$ sudo aptitude install apache2-dev php5-dev build-essential
# get current version of apc - check if there is a newer one!
$ wget http://pecl.php.net/get/APC-3.0.16.tgz
$ tar xzf  APC-3.0.16.tgz
$ cd  APC-3.0.16
$ phpize
$ ./configure --enable-apc --enable-apc-mmap \
  --with-apxs=/usr/bin/apxs2 \
  --with-php-config=/usr/bin/php-config
$ make
$ sudo make install
# in /etc/php5/apache2/php.ini add: extension=apc.so
$ sudo apache2ctl restart

No a phpinfo(); should show you a new apc section.