Recovering Linux file permissions

I recently ran into a server, where somebody accidently issued a „chown -R www-data:www-data /var“. So all files and directories within /var where chowned to the www-data which actually means a complete system fuckup as everything from logging over mail and caching to databases relies on a correct setup there. Sadfully this was a remote production server so I had to find a quick solution to get a least a state good enough for the next days.

I started peaking around a possibity to reset file permissions based on .deb package details. There are at least approaches (the method there misses a pre-download of all installed .deb packages) to do this (and I remember running a program years ago that checked file permissions based on .deb files – just did not find it via apt-get). Nonetheless this approach lacks the possibility of handling application created files. Files in /var/log for instance don’t have to be declared in a .deb file but urgently need the right file permissions.

So I came to a different approach: cloning permissions. By chance we had a quite similar server running meaning same Linux distribution and nearly the same services installed. I wrote a one liner to save the file permissions on the healthy server:

$ find /var -printf "%p;%u;%g;%m\n" > permissions.txt

The command writes a text file with the following format:


Please note, I started using „:“ as a separator but noted that at least some Perl related files have a double colon in there name.

Now I only needed a simple shell script that sets the file permissions on the broken server based on the text file we just generated. It came down to this:


ENTRIES=$(cat permissions.txt)

	echo ${ENTRY} | sed "s/;/ /g" | {
		chown ${USER}:${GROUP} "${FILE}"
		chmod ${MODE} "${FILE}"

The script reads every line of the text file, splits it’s content into variables and sets the user and group via „chown“ as well as the mode via „chmod“. It doesn’t check if a directory/file exists before chowning/chmodding it, as it actually doesn’t matter. If it’s not there, it just won’t do something harmfull.

After you’ve run this, it’s a good idea to restart all services and start watching log files. You have to take care of all services that rely on fast changing files in /var. For instance a mail daemon puts a lot of unique file names into /var/spool and the script above won’t be able to take care of that. You have to double check database directories like /var/lib/mysql, hosted repositories and so on. But the script will provide with a state where most services are at least running and you get an idea of how to switch back the remaining directories. It might be helpfull to search for suspicious files, like

$ find /var -user www-data

my package of the day – gpg for symmetric encryption

Though asymmetric encryption is state of the art today, there are still cases when you probably are in need of a simple symmetric encryption. In my case, I need an easy scriptable interface for encrypting files for backup as transparent as possible. While you can, of course, use asymmetric encryption for this, symmetric methods can save you a lot of time while still being secure enough.

So there are methods like stupid .zip encryption or a bunch of packages in the repositories like „bcrypt“ that provide you with their implementations. But there is a tool, you already know and maybe even use, but don’t think of when considering symmetric encryption: „gpg„. Actually gpg heavily relies on symmetric algorithms as you might know. The public/private key encryption is a combination of asymmetric and symmetric encryption as the latter is quite more cpu efficient. In our case, gpg will use the strong cast5 cipher by default.


So as gpg already knows about a bunch of symmetric encryption algorithms, why not use them? Let’s just see an example. You have a file named „secretfile1.txt“ and want to encrypt it:

$ gpg --symmetric secretfile1.txt

You will be prompted for a password. Afterwards you’ll have a file named secretfile1.txt.gpg. You are already done! Please note: The file size of the encrypted file might have decreased as gpg also compresses during encryption and outputs a binary. In my test case the file size went down from 700k to 100k. Nice.


In case you need to have an easy portable file that is even ready to be copy-pasted, you can ask gpg to create an ascii armor container:

$ gpg --armor --symmetric secretfile1.txt

The output file will be called „secretfile1.txt.asc“. Don’t bother to open it in a text editor of your choice. The beginning will look similar to this:

$ head secretfile1.txt.asc
Version: GnuPG v1.4.6 (GNU/Linux)


(In this case I used ‚head“ for displaying the first ten lines. Head is similar to tail, which you might already know.) Though the ascii file is larger than the binary .gpg file it is still much smaller than the original text file (about 200k in the above case). When tampering with binary files like already compressed tarballs the file size of the encrypted file might slightly increase. In my test, the size grew from a 478kB file to a 479kB file when using binary mode. In ascii armor mode the size nearly hit the 650kb mark, which is still pretty acceptable.


Decrypting is as easy. Just call „gpg –decrypt“, for instance:

$ gpg --decrypt secretfile1.txt.gpg
# or
$ gpg --decrypt secretfile1.txt.asc

gpg knows by itself, if it is given an ascii armored or binary file. But nevertheless the output will be written to standard output, so the following line might be much more helpful for you:

$ gpg --output secretfile1.txt --decrypt secretfile1.txt.gpg

Please note, that you need to stick to the order first –output, then –decrypt. Of course you can also use a redirector („>“).


So, for the sake of it: The real interesting thing is that you can use gpg symmetric encryption in a chain of programs, controlled by pipes. This enables you to encrypt/decrypt on the fly with shell scripts helping you to write strong backup scripts. Gpg already detects, if your are using it in a pipe. So let’s try it out:

$ tar c directory | gpg --symmetric --passphrase secretmantra \
| ssh hostname "cat > backup.tar.gpg"

We just made a tarbal, encrypted it and sent it over ssh without creating temporary files. Nice, isn’t it? To be honest, piping over ssh is not a big deal anymore. But piping to ftp? Check this:

$  tar c directory | gpg --symmetric --passphrase SECRETMANTRA \
| curl --netrc-optional --silent --show-error --upload-file - \
--ftp-create-dirs ftp://USER:PASSWORD@SECRETHOST/SECRETFILE.tar.gpg

With the mighty curl we just piped from tar over gpg directly to a file on a ftp server without any temporary files. You might get a clue of the power of this chain. Now you can start using a dumb ftp server as encrypted backup device now completely transparently.

That’s all for now. If you like encryption, you should also check symmetric encryption and the possibilities of enhancing daily business scripts security by adding some strong crypto to it. Of course you can complain about the security of the password, the possible visibility via „ps aux“, but you should be able to reduce risks by putting some brain in it. In the meantime check „bashup„, the bash backup script, which uses methods described here to provide you with a powerful and scriptable backup library written in Bash with minimum depencies. And yes, gpg will be added soon.

my package of the day – sash – the Stand Alone SHell for system recovery

Let me introduce you today to a package that is quite unknown as you hopefully never need it. But when you need it and have not thought about it before, it is probably already too late. I am talking about „sash“ – the „Stand Alone SHell“. Yet another shell? Yes and no. Yes it is a shell, but no, I am not trying to show something like the shiny friendly interactive shell or (my favorite) „zsh“. Quite the contrary: You can give „sash“ a lot of attributes, but not „shiny“.

So what is about? Imagine the following case: You are running a machine and suddenly something goes totally wrong. Partition errors, missing libraries, you have messed around with libc, whatever. This can get you into serious trouble. You are fine, when you have the possibility to boot a recovery cd or something similar. But under some circumstances you might have to stick to the programs already installed though they seem to be broken. Maybe it is a virtual server somewhere on the web and you are only allowed to boot into a recovery mode giving you a prompt to your server. So you are trying to login as root but it just does not work for some reasons. Broken dependencies. Who knows.

The point is: When you login onto a machine for system recovery, you are already relying on a lot of tools and dependencies – though it only seems to be a shell. The shell might be linked against a couple of libraries, a lot of commands you want to run are not build in and therefore a bunch of external dependencies can bar your way. So what you actually might need in a situation of severe pain is a shell that provides you with as much essential tools as you need on its own without relying on external code.

Installing sash

This is where „sash“ comes into play. Sash is not a dynamic linked executable, it has actually all needed features built in. So as long as you can execute the sash binary, you can have a working shell. Let’s check it! Install „sash“ by using „aptitude install sash“ or you preferred package manager. Please note, that sash will clone your current root account:

cloning root account entry to create sashroot account in /etc/passwd
Cloned sashroot from root in /etc/passwd
cloning root account entry to create sashroot account in /etc/shadow
Cloned sashroot from root in /etc/shadow

So you have this new line in your /etc/passwd:


You should consider giving sashroot a password if you want to be able to login with this account. But please check if this applies to your security needs.

See the difference

Now let’s check how the sash binary differs from the standard shell, the bash and the zsh. We are using „ldd“ for this, as it is lists libraries, an executable is linked against:


Pretty impressive. All „normal“ shells have at least three dependencies, sash apparently has none.

But getting rid of external libraries is not the only difference sash makes. Another major feature is the collection of built-in commands:

-ar, -chattr, -chgrp, -chmod, -chown, -cmp, -cp,
-dd, -echo, -ed, -grep, -file, -find, -gunzip,
-gzip, -kill, -ln, -ls, -lsattr, -mkdir, -mknod,
-more, -mount, -mv, -printenv, -pwd, -rm, -rmdir,
-sum, -sync, -tar, -touch, -umount, -where

Seems like a list of commands you yearn for, when in recovery mode, don’t you? Note the leading „-“ at the beginning of those commands. This is the way, sash handles your attempts to run internal and external commands. When using „mv“, sash gives you the normal /bin/mv, when using „-mv“, sash provides you with it’s own replacement. But „sash“ helps you when you don’t want to type the „-“ at the beginning of every command. You can enter „aliasall“ in a sash session as it will create non permanent aliases for all builtin commands:



In case of an emergency you might need to boot directly into sash as maybe your initrd stuff is broken. How? Just append a „init=/bin/sash“ to your kernel command line – be it lilo or grub. This way, you will be directly dropped into a sash session.

What’s missing?

Sadfully one essential command is missing: fsck. As the sash manual points out: fsck is just way too big to be included in a statically linked binary. Sad, but true. But hey: Better being to able at least to act on the console than having no console at all.

Sash as a standard shell?

… is not a good idea. It just lacks a lot of features you’ll really want when working on the command line: A verbose prompt, command history, tab completion and so on.

So it’s to install sash now as you will miss it, when it’s too late :)
(And just if you’d like to ask: Yes, at least once I really needed sash and it helped me to save a lot of time.)

my package of the day: file – classify (unknown) files and mime-types on the console

You know this? Somebody just sent you a mail with attachments that don’t have usable file extensions so you don’t really know how to handle them. Audio file? PDF? What is it? The same problem might occur after a file recovery, on web pages with upload features or just when you are really and time pressure and have time for messing around with file type guessing.

While you can try to give the file an extension and open it with a software you think might be suitable, the more sophisticated way is to let your computer find out what is all about. As a GNU/Linux user you probably already think „There is surely a command line tool for this“. Of course there is: The package „file„, that often gets automatically installed by dependencies or just an „aptitude install file“ will help you out.

„file“ depends on „libmagic“ which provides patterns for the so called „magic number“ detection. You don’t have to know, what that is, but if you want, see this Wikipedia article for reference. So all you have to know, is how to handle the file command. And actually there is not much to learn. Let’s assume we have the following directory with unknown files:


Now we want to know what’s inside those black boxes. Therefore we just call „file *“ on the console:


Hey, that’s all. Pretty impressive, isn’t it? „file“ does even not only differs binary from text files, it even tries to guess what programming language a text file is written in. And the magic is not that much magic: In case of the zsh file it just sees a shebang pointing to the zsh in the first line of the file, a PDF file typically starts with „%PDF“ and so on. It’s all about patterns.

„file“ provides you with some command line options that make it’s usage even more helpful. The most interesting is „-i“ as it prints out mime types instead of verbose file types. If you are a web developer and want to know the exact mime type for a file download, this can save you a lot of time:


Great, isn’t it? The Apache webserver also uses libmagic for this purpose. With „file“ you just use a wrapper for the same task.

That’s all about „file“ for today. Happy file detection – and feel free to report back.

my package of the day: listadmin – moderate mailman mailing lists from the console

Are you involved in moderating Mailman mailing lists? Then maybe you know the pain: As you try to stop spammers flooding you list you hold messages from unknown senders back for review. Or you have a moderated mailing list that only allows postings explicitly published. However. In most cases you get mails from Mailman telling you that there are messages you have to moderate. The common way is to enter the web interface, enter your password, read the messages and discard/reject/allow them.

This workflow is easy but it can really get on your nerves as the web stuff is somehow time consuming. Therefore from time to time you get lazy on moderating…

Well, there is light at the end of the tunnel and one cannot repeat this hint often enough: The package „listadmin“ provides a powerful console tool for moderating mailman mailing lists. As Debian/Ubuntu user you just have to install it via „aptitude install listadmin“ on the console or via Synaptic. You just have to write an .ini file with the configuration (admin url, credentials). The files looks like this:

adminurl https://hostname.tld/mailman/admindb/{list}
default skip
log ~/.listadmin.log
password secret
[email protected][email protected]

So we just give an url with a placeholder named „{list}“. This way we can moderate multiple lists with fewer lines of configuration. Now let’s see how listadmin behaves when checking existing mailing lists (in this case our Berlin based Ubuntu mailing lists):


Nothing to do – no messages to moderate in this case. But hey – we just got an incoming request. Let’s rerun listadmin and check:


A spammer tried to hit our list. We now can decide wether to Approve, Reject or Discard the message. If it’s spam you want to discard as this just deletes the message. When you want to provide feedback to the user, you have to reject and are able to enter a reason. Of you course you also can examine the full body of the message or just skip it and keep for the next session. In our case „d“ was entered for deleting the spam and the request was submitted. If you are fast the session will not take more than 10 seconds – try this with the web interface!

So though it’s age and the ajax web 2.0 shiny wysiwyg plinkplonk alternatives Mailman provides you with nice wrappers for moderating larger amounts of mails within seconds. If you stick to a community you will probably sooner or later be asked to moderate a mailing list. Now you can say: „No problem. I have a command line tool for this“.

my package of the day: fish – the friendly interactive shell

Always wanted to learn using a shell more deeply? Maybe „fish„, the „friendly interactive shell“ is the right kickoff for you.

If you are already a heavy command line user with customized .bashrc or even .zshrc (like me), thank you probably don’t need another shell. But if this shell thingy is somehow a miracle to you but you saw people using it like wizards with colorful commands and a typing speed that made you jealous then it could help you to start with a shell that concentrates on being very friendly to new users as common shells like Bash and ZSH expect you to read the manual and write a config file (there are aids and defaults that vary from distribution to distribution).

The standard shell for login users in Ubuntu/Debian is „Bash“. Ubuntu already ships the file /etc/bash_completion that is read by default and helps users using the TAB key more exensively. Try it on you bash shell: just type something like „ls –“ and press TAB twice. You’ll see a list of options that „ls“ provides. Nice but it could be nicer. Let’s compare this to fish. Install fish by using Synaptic or „aptitude install fish“, open a terminal and start the shell by typing „fish“. You should a changed green prompt. Now type „ls -“ and press TAB.

Stop: Already while typing you should see a strange color change. When entering „l“ the character turns red and underlined. Looks like an error? Well, it is: fish tells you, that „l“ is probably not a command. An aid during typing before running a command. Neat. Now, when pressing TAB you should a very clean list of options for „ls“ with a short description of each option:


Helpfull, isn’t it? Of course this is not limited to ls. Try it with other commands you are using. If you ask yourself why you have to type „command –“ and press TAB: „–“ introduces a command line option („-“ does this also – try it!). As you press TAB after this, the shells knows „the user wants to do something and needs help on completing it“. It looks after a pattern and sees that you want to use the given command and are looking for options. That’s all. As I said: This works in Bash often by default also, but not that nice.

Now fish can do more with completion of course. Want to install a program? Try „aptitude install mut“ and press TAB. It will show you a list of packages matching that pattern:


Need to kill a process? Type „kill “ and press TAB and you will get a nice list of running processes:


The list of possible TAB completions on fish is endless. Just notice that emphasis has been put on commands like mount, make, su, ssh, apt-get/aptitude. In most commands usernames, process ids will automatically be completed. The trick is just to try TAB when you are too lazy to type or unsure how to proceed. A good shell surprises you from time to time with it’s completion.

Also very helpful is the extended pattern matching for file names. Let’s say you want a list of all pdf files in a directory and all it’s subdirectories. On bash you probably use something like „find . -name „*.mp3“. On fish you use the pattern „**“ which means any files and directories in the current directory and all of its subdirectories. So type „ls **.pdf“ and you get the list you want as fish crawls through the directories for you. Want alle .mp3 and mp4 files but not files like .mpeg? Use „ls **.mp?“ as „?“ stands for one character. Of course commands like „rm **.bak“ are possible, too. Use them with care! In the following example we are looking for pdf files in all subdirectorie, delete them and afterwards make sure they are really gone:


So let me stop here. I hope, I was able to show you that using fish instead of an unconfigured shell is a nice way of getting in the command line business. Fish provides you with a lot of more features that you might need and saves you from writing a config file from scratch.

If you want to give fish a try: Install it and run the „help“ command. I will launch a nice help page in you browser. Read some parts of the document as they’ll show you nice gimmicks. Or just don’t and start right away. But trust me: Reading hints for a shell from time to time will save you … time.

(Just in case you don’t know: You can change your standard shell by using the „chsh“ command. But when being a novice it is always a good idea to stick to the distribution specific default shell and run your shell directly by calling it. When you are more used to it feel free to make it your standard shell…)

my package of the day: weather-util (weather report and forecast for the console)

Let me introduce you today into a tool that a lot of people might evaluate as useless: Jeremy Stanley’s weather-util. Whith this tiny python script, which finally found its way into Debian Etch and Ubuntu repositories, you can retrieve weather information from weather stations worldwide directly from the command line.

After installing it by running „aptitude install weather-util“ or synaptec, call „weather“:

$ weather
Current conditions at Raleigh-Durham International Airport (KRDU)
Last updated Jun 04, 2008 - 01:51 AM EDT / 2008.06.04 0551 UTC
   Wind: from the S (180 degrees) at 10 MPH (9 KT)
   Sky conditions: mostly cloudy
   Temperature: 72.0 F (22.2 C)
   Relative Humidity: 73%

Pretty impressive, isn’t it? Weather just makes an http call to a weather server for a preset station (where the heck is Raleigh-Durham International Airport?) and returns the current weather information. Of course you can also retrieve the forecast for the next days by running „weather -f“:

$ weather -f
Current conditions at Raleigh-Durham International Airport (KRDU)
Last updated Jun 04, 2008 - 01:51 AM EDT / 2008.06.04 0551 UTC
   Wind: from the S (180 degrees) at 10 MPH (9 KT)
   Sky conditions: mostly cloudy
   Temperature: 72.0 F (22.2 C)
   Relative Humidity: 73%
City Forecast for Raleigh Durham, NC
Issued Wednesday morning - Jun 4, 2008
   Wednesday... Partly cloudy, high 67, 20% chance of precipitation.
   Wednesday night... Low 96, 20% chance of precipitation.
   Thursday... Partly cloudy, high 71, 10% chance of precipitation.
   Thursday night... Low 97.
   Friday... High 72.

Sadfully the forecast only displays Fahrenheit, but that way we have enough space for patching the package :)

Retrieving local weather information

Now we are, of course, we are interested in the weather in our area. The easiest way is getting the ID for a weather station. Just go to and choose your country/city/station by using the drop down menus for US and international stations. When you found a station close to your point of interest you can see a four letter id in round brackets. See the example above – the airport has KRDU. I am using EDDI most of the times which is Berlin Tempelhof – an airport in the city center of Berlin.

So you are ready to ask politely for weather again by giving the id with „weather –id=ID“, in my case „–id=EDDI“. (note: you can also make it short with „-iEDDI“:

$ weather --id=EDDI
Current conditions at Germany (EDDI) 52-28N 013-24E 49M (EDDI)
Last updated Jun 04, 2008 - 01:50 AM EDT / 2008.06.04 0550 UTC
   Wind: from the E (080 degrees) at 13 MPH (11 KT)
   Temperature: 62 F (17 C)
   Relative Humidity: 59%

Please note: Not all weather stations support forecasts (-f) and drop a 404 http error. You just have to try this. You can also switch on „verbose“ mode (-v) which gives you even more details.

Weather on the command line without weather-util?

Works like a charm, doesn’t it? For the curious people around who want to understand where weather-util pulls the information from: See

for reference. Just text files on a web server regularly updated. Click around and go to there parent dir – you’ll find even more interesting information. So using weather-util without weather-util should be not a big deal.

Screen integration

Now for the console lovers: You are using screen with a pimped status bar, don’t you? And in your wildest dreams you imagined the status bar showing the weather report, so you even don’t have to look outside the window because as a console guy you don’t even like your real „window“? No problem anymore by using screens backticks and weather-util.

As I noticed that weather-util runs into trouble from time to time when not being able to send it’s http request, I decided for a indirect weather pull by writing the information I need to a flat file by a cronjob. We just call weather-util and use awk to grab the snippet we need. I am interested in the temperature in Celsius. weather-util shows this line:

Temperature: 62 F (17 C)

So I use the following very quick and very dirty awk to get the „17“ out:

$ weather -iEDDI | awk '/Temperature/ {print $4}' | \
awk -F "(" '{print $2}'

Feel free to brush this up and report back. I am sure you can improve to use only one awk call instead of two.

You save this line to a shell script that is scheduled to run every five minutes and direct it via „>“ to write it’s output to a flat txt file. Within you .screenrc you read this file and display the contents in you status bar.

startup_message off
defscrollback 1024
hardstatus on
hardstatus alwayslastline
backtick 1 0 300 cat /path/to/weather-text-file.txt

# remove line breaks made with "\"on the following lines
caption always "%{+b rk}$USER@%{wk}%H | %{yk}(Last: %l) %{gk} \
Weather: %1`C  %-21=%{wk}%D %d.%m.%Y %0c"
hardstatus alwayslastline "%?%-Lw%?%{wb}%n*%f %t%?(%u)\
%?%{kw}%?%+Lw%? %{wk}"

Make sure that have the file /path/to/weather-text-file.txt with the temperature in it. Now run screen and enjoy you shiny new status bar. See the green area in the screenshot below:

So that’s all for now. You should be able to play around with weather-util and screen to get the information you need (or let’s say „want“ :).


The incredible mnemonikk updated my awk | awk to a onetime sed within seconds:

$ weather -iEDDI | sed -n 's/.*Temperature:.*(\(.*\))/\1/p'

Thank you!

good howto: Bash Pitfalls

There is a very nice collection of common Bash scripting pitfalls and hints on how to avoid them on Greg’s Wiki: Bash Pitfalls

If you are writing little Bash scripts from time to time or are even a heavy Bash scripter, give it a try – it helps you avoiding errors that might work perfectly under normal circiumstances but suddenly go wrong… A good guide especially for writing server bullet proof scripts.

new kernel release detection snippet

Just a small and old snippet that might be helpful or an example: Some years ago I’s in need of getting to know early about new released Linux kernel versions. Therefore I wrote a (not sophisticated but working) crontabbed script checking the kernel page for a new stable Linux kernel and alerting me via mail if a new version is found with link to the changelog:

CURRENTVERSION=`w3m -dump \ \
 | head -n 1 | awk '{print $10}'`
SAVEDVERSION=`cat ~/bin/kernelversion.log \
 | tail -n 1 | awk '{print $2}'`
SAVEDDATE=`cat ~/bin/kernelversion.log \
 | tail -n 1 | awk '{print $1}'`
  CURRENTDATE=`date +'%Y-%m-%d'`
  >> ~/bin/kernelversion.log
  echo -e "Detected new kernel version \
   (replacing version ${SAVEDVERSION} from\
   ${SAVEDDATE}). Please check \\
   ${CURRENTVERSION} forr details." \
 | mail -s "new kernel ${CURRENTVERSION}" \

The only real bug in this script is that it does not detect network issues and therefore alerts you when it is not able to get a http response. But this could be fixed with one or two lines of code. And yes most lines could be more elegant :) Probably today there are better channels like rss or even an old mailing list with announcements that I never looked for, but this snippet does it’s job very well.


Fixed the broken wrapping of the script. Sorry about this. (Thank you Jeremy.)

Jonne stated that of course using a feed like is the better choice today. He is surely right about this  though sometimes receiving a mail is a need.