Recovering Linux file permissions

I recently ran into a server, where somebody accidently issued a “chown -R www-data:www-data /var”. So all files and directories within /var where chowned to the www-data which actually means a complete system fuckup as everything from logging over mail and caching to databases relies on a correct setup there. Sadfully this was a remote production server so I had to find a quick solution to get a least a state good enough for the next days.

I started peaking around a possibity to reset file permissions based on .deb package details. There are at least approaches (the method there misses a pre-download of all installed .deb packages) to do this (and I remember running a program years ago that checked file permissions based on .deb files – just did not find it via apt-get). Nonetheless this approach lacks the possibility of handling application created files. Files in /var/log for instance don’t have to be declared in a .deb file but urgently need the right file permissions.

So I came to a different approach: cloning permissions. By chance we had a quite similar server running meaning same Linux distribution and nearly the same services installed. I wrote a one liner to save the file permissions on the healthy server:

$ find /var -printf "%p;%u;%g;%m\n" > permissions.txt

The command writes a text file with the following format:

dir/filename;user;group;mode

Please note, I started using “:” as a separator but noted that at least some Perl related files have a double colon in there name.

Now I only needed a simple shell script that sets the file permissions on the broken server based on the text file we just generated. It came down to this:

#!/bin/bash

ENTRIES=$(cat permissions.txt)

for ENTRY in ${ENTRIES}
do
	echo ${ENTRY} | sed "s/;/ /g" | {
		read FILE USER GROUP MODE
		chown ${USER}:${GROUP} "${FILE}"
		chmod ${MODE} "${FILE}"
	}
done

The script reads every line of the text file, splits it’s content into variables and sets the user and group via “chown” as well as the mode via “chmod”. It doesn’t check if a directory/file exists before chowning/chmodding it, as it actually doesn’t matter. If it’s not there, it just won’t do something harmfull.

After you’ve run this, it’s a good idea to restart all services and start watching log files. You have to take care of all services that rely on fast changing files in /var. For instance a mail daemon puts a lot of unique file names into /var/spool and the script above won’t be able to take care of that. You have to double check database directories like /var/lib/mysql, hosted repositories and so on. But the script will provide with a state where most services are at least running and you get an idea of how to switch back the remaining directories. It might be helpfull to search for suspicious files, like

$ find /var -user www-data

RubyGems 9.9.9 packaged – Fake install RubyGems on Debian/Ubuntu

For a lot of reasons I often rely on a mixture of a Debian/Ubuntu pre packaged Ruby with a self compiled RubyGems. It helps you in situations where you don’t care that much about the Ruby interpreter itself but need an up to date RubyGems. While this is easy to install, you might run into trouble when installing packages that depend on Ruby and RubyGems, namely packages like “rubygems”, “rubygems1.8″ and “rubygems1.9″.

After unsuccessfully playing around with dpkg for a while (you can put packages on “hold” which prevents them from being installed automatically, I came to the conclusion, the best way is to install a fake package that is empty but satisfies depencies.

So, here it is: The shiny new RubyGems 9.9.9 which delivers rubygems, rubygems1.8 and rubygems1.9 right away. Just install it (e.g. with dpkg) and you’ll be able installing packages that rely on a rubygems package.

In case you want to play around with the package and customize it to your needs, e.g. only deliver rubygems1.8 or rubygems1.9, take

1. Install equivs

$ sudo apt-get install equivs

2. create a control file

$ equivs-control rubygems

3. edit the control file

$ vim rubygems

You can compare the default settings in the control file with the output of e.g. “apt-cache show rubygems”. The crucial field is “Provides:” where you can put a comma separated list of packages you want to fake install. Choose a high version for  there “Version: ” field as this will mark the package newer as the distribution’s own package. This prevents the packager from replacing it.

Section: universe/interpreters
Priority: optional
Homepage: http://www.screenage.de/blog/
Standards-Version: 3.6.2
 
Package: rubygems
Version: 9.9.9
Maintainer: Caspar Clemens Mierau <ccm@screenage.de>
Provides: rubygems1.8,rubygems1.9,rubygems
Architecture: all
Description: Fake RubyGems replacement
 This is a fake meta package satisfying rubygems dependencies.
 .
 This package can be used when you installed a packaged ruby but want
 to use rubygems from source and still rely on software that depends
 on ruby and rubygems

4. build the package

$ equivs-build rubygems

p.s.: You can also use equivs for easily building meta packages containing a list of packages you want to install at a glance, e.g. for semi automated server bootstrapping.

Bootstrapping a Puppet agent/master on Ubuntu

Though it’s really great that Puppet made it into Ubuntu’s main repository, the provided version is rather outdated which prevents you from using advanced language features when writing your manifests. So sooner or later you end up installing Puppet manually. In order to speed up installation I stripped it down to the following:

install agent:

$ bash &lt; &lt;(wget -qO - https://bit.ly/install-puppet-agent)

install master:

$ bash &lt; &lt;(wget -qO - https://bit.ly/install-puppet-master)

The call fetches the most recent version of the install script from github, installs Ubuntu’s Ruby (which is good enough for running Puppet), fetches an upstream version of gem itself and updates it to the most recent version and finally installs the Puppet gem.

You can, of course, also download, review and run the scripts manually. Just have a look at https://github.com/moviepilot/puppet/tree/master/tools

slides from the ‘From MySQL to MariaDB’ presentation

As announced, I held a short talk on switching from MySQL community edition (especially 5.1) to MariaDB (currently 5.2.6) at this years LinuxTag in Berlin.

Here are the (German) slides for reference:

(In case you cannot see the embedded presentation, you can also click here)

Please note: There are a lot of good English slides around. If you want give a talk on MariaDB, the “Beginner’s Guide” might be a good start:

A Beginner’s Guide to MariaDB Presentation

Short talk on MariaDB at Linuxtag 2011

If you happen to be around at this years LinuxTag 2011 in Berlin/Germany, you are invited to attend my short talk on MariaDB as a drop-in replacement for MySQL. The talk focusses on differences between MySQL Community Edition and MariaDB (e.g. XtraDB, Aria, userstats), shows some features live and explains how to switch. I’ll probably post the slides here afterwards.

The talk will be held in German and is scheduled for Friday, the 13th of May, 16:30. The official announcement can be found here.

Using backuppc as a dirty distributed shell

Backuppc is a neat server-based backup solution. In Linux envorinments it is often used in combination with rsync over ssh – and, let’s be hontest – often fairly lazy sudo or root rights for the rsync over ssh connection. This has a lot of disadvantages, but at least, you can use this setup as a cheap distributed shell, as a good maintained backuppc server might have access to a lot of your servers.

I wrote a small wrapper, that reads the (especially Debian/Ubuntu packaged) backuppc configuration and iterates through the hosts, allowing you to issue commands on every valid connection. I used it so far for listing used ssh keys, os patch levels and even small system manipulations.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
#!/bin/bash
SSH_KEY="-i /var/lib/backuppc/.ssh/id_rsa"
SSH_LOGINS=( `grep "root" /etc/backuppc/hosts | \
 awk '{print "root@"$1" "}' | \
 sed ':a;N;$!ba;s/\n//g'` )
 
for SSH_LOGIN in "${SSH_LOGINS[@]}"
do
 HOST=`echo "${SSH_LOGIN}" | awk -F"@" '{print $2'}`
 echo "--------------------------------------------"
 echo "checking host: ${HOST}"
 ssh -C -qq -o "NumberOfPasswordPrompts=0" \
 -o "PasswordAuthentication=no" ${SSH_KEY} ${SSH_LOGIN} "$1"
done

You can easily change this to your needs (e.g. changing login user, adding sudo and so on).

$ ./exec_remote_command.sh "date"
--------------------------------------------
checking host: a.b.com
Mo 9. Mai 15:40:26 CEST 2011
--------------------------------------------
checking host: b.b.com
[...]

Make sure to quote your command, especially when using commands with options, so the script can handle the command line as one argument.

A younger sister of the script is the following ssh key checker that lists and sorts the ssh keys used on systems by their key comment (feel free to include the key itself):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#!/bin/bash
 
SSH_KEY="-i /var/lib/backuppc/.ssh/id_rsa"
SSH_LOGINS=( `grep "root" /etc/backuppc/hosts | \
 awk '{print "root@"$1" "}' | \
 sed ':a;N;$!ba;s/\n//g'` )
 
for SSH_LOGIN in "${SSH_LOGINS[@]}"
do
 HOST=`echo "${SSH_LOGIN}" | awk -F"@" '{print $2'}`
 echo "--------------------------------------------"
 echo "checking host: ${HOST}"
 ssh -C -qq -o "NumberOfPasswordPrompts=0" \
 -o "PasswordAuthentication=no" ${SSH_KEY} ${SSH_LOGIN} \
 "cut -d: -f6 /etc/passwd | xargs -i{} egrep -s \
 '^ssh-' {}/.ssh/authorized_keys {}/.ssh/authorized_keys2" | \
 cut -f 3- -d " " | sort
 ssh -C -qq -o "NumberOfPasswordPrompts=0" \
 -o "PasswordAuthentication=no" ${SSH_KEY} ${SSH_LOGIN} \
 "egrep -s '^ssh-' /etc/skel/.ssh/authorized_keys \
 /etc/skel/.ssh/authorized_keys2" | cut -f 3- -d " " | sort
done

A sample output of the script:

$ ./check_keys.sh 2>/dev/null
--------------------------------------------
checking host: a.b.com
ccm@host1.key
backuppc@localhost
some random key comment
--------------------------------------------
checking host: b.b.com
[...]

That’s all for now. Don’t blame me for doing it this way – I am only the messenger :)

Ubuntu (Berlin) Global Jam at c-base and Daniel Holbach’s notebook

Members of “Ubuntu Berlin” met yesterday at c-base within the Ubuntu Global Jam. While it was nice seeing new and international faces showing up and introducing newcomers to advanced Launchpad usage, my main attraction of the day was Daniel Holbach’s notebook. He asserted it runs Maverick and starts up within five seconds, which made me laugh at first as my netbook’s startup time tripled from Lucid to Maverick to round about 45 seconds (which will at least change back until release I assume).

Ubuntu Berlin at Ubuntu Global Jam (c-base) - August 2010

Ubuntu (Berlin) Global Jam at c-base

To make it short: Between bug triaging and patching Daniel showed the startup procedure two or three times on his X61s (with an solid state disk, one has to add) and as promised it started up in five seconds after Grub. Actually this isn’t more than a fast booting notebook, but it shows the results of focussed efforts from the last one and a half year. Remember the initial “10s” posting and the bunch of changes it took.

So I am happy looking forward to improvements for Maverick on my netbook. And yes: I am happy with 10 seconds, too.

[update]

Daniel noted, that it’s a X61s, not a T61. Changed.

Desktop Summit 2011 in Berlin

I am happy to announce that Berlin has been chosen as location for the Desktop Summit 2011. If you don’t know so far: Desktop Summit is a 1000+ developer conference co-hosting KDE’s “Akademy” and GNOME’s “GUADEC” at the same time:

Read the press release: Desktop Summit 2011 Announced

As Ubuntu member and head member of c-base e.V. I am part of the Berlin team, together with Claudia Rauch from KDE e.V. and Mirko Boehm of KDE. Let me quote Mirko:

“We are honored and proud that our proposal was selected. What we look forward to the most is the inspiration our communities will draw from having the Desktop Summit together again, but also from visiting our bustling, welcoming city. We would like to thank all the supporters of the proposal, and will work hard to make the conference a big success.”

I am sure this event will become a success. And it’s a great opportunity to meet and greet across the letters before the “U” in “Ubuntu”.

Ubuntu Berlin @ LinuxTag 2010 – pickings

Saturday evening this years LinuxTag, Europes largest open source fair, closed its doors. As LinuxTag is presented at Berlin, the “Ubuntu Berlin” was happy to support it in different activities. Let me sum up some of them:

1) Ubuntu Community booth

The “ubuntu Deutschland e.V.”, Ubuntu and Kubuntu community presented their work at a community hosted booth. A bunch of Ubuntu Berlin members supported the booth, answered hundreds of questions and helped with the proceedings.

2) Talks

Saturday Featured a lot of Ubuntu focussed talks. For the first time “Ubuntu Berlin” and its members hosted a remarkable amount of them, e.g.:

  • Anselm Helbig – Ubuntu Berlin Lightning Talks (tmux)
  • Benjamin Drung – Ubuntu in 50 minutes, Packaging for Debian and Ubuntu
  • Caspar Clemens Mierau – Ubuntu Berlin Lightning Talks (Vimperator), Gnome-Do, Ubuntu in 50 minutes
  • Daniel Holbach – Fixing Bugs in Ubuntu (see slides), Ubuntu in 50 minutes, Ubuntu Berlin Lightning Talks (liblaunchpad)
  • Marcel Eichner – Ubuntu Berlin Lightning Talks (Franklin)
  • Torsten Franz – Supporting Ubuntu

Ubuntu in 50 minutes talk at LinuxTag 2010
minutes before the “Ubuntu in 50 minutes” talk

3) Interviews for Radio Tux

Several members of Ubuntu Berlin gave interviews to the well known German Podcast “Radio Tux“. Daniel’s talk on bugjamming has been released already. Be sure to check the archive for other releases within the next days.

4) LinuxTag BBQ

Last but not least: The  ”End of LinuxTag BBQ” at c-base, sponsored by Canonical, has been a great success again. Three barbacues and about ten “Grillmeister” (you cannot translate this) provided more than hundred visitors with tasty food. Members of various types of open source communities had interesting chats while relaxing at the banks of the Spree.

As Ubuntu Berlin’s support for the LinuxTag continuously increased within the last years, I am sure next year will even better. We’ll see.

Ubuntu Berlin LinuxTag BBQ today

For those of you currently staying in Berlin for the LinuxTag 2010: I know where you are going today evening after the fair closes. For the third time Ubuntu Berlin hosts a LinuxTag BBQ at c-base, the sunken starbase in the heart of Berlin. Be sure to grab free food there – friendly sponsored by Canonical (thank you!) and a refreshing botte of beer or the beloved Club Mate at the bar for reasonable prices.

The BBQ opens around 6pm. If you don’t know how to get to c-base from the LinuxTag area, a friendly team will guide you – just have a look at the official announcement. The last two your showed, the LinuxTag BBQ is a perfect ending as a large crowd of open source developers and community members of all free project types take the possibility to chat and relax directly at the riverside of the Spree. Don’t miss that :) See you there.