Transmission daemon: remove finished torrents

In transmission-daemon, downloaded torrents stay listed as long as they are not manually removed. In order to get them flushed automatically, the following command run by cron can help:

transmission-remote -l | grep 100% | grep Done | \
awk '{print $1}' | xargs -n 1 -I % transmission-remote -t % -r

In order to add this to cron, the "%"-symbols have to be escaped with a leading slash like this:

*/5 * * * * /usr/bin/transmission-remote -l | grep 100\% | grep Done | awk '{print $1}' | xargs -n 1 -I \% /usr/bin/transmission-remote -t \% -r

FlexGet Installation in Ubuntu

*[PyPI]: Python Package Index

This post is deprecated. Please go to http://www.grosseosterhues.com/2012/02/flexget-installation-and-upgrade/ for more recent information.

This posts documents the installation of FlexGet in Ubuntu. It uses FlexGet v1.0r2301 and by the time of writing it is tested on Ubuntu 11.04.

There is no apt-package for FlexGet available yet. It has to be installed from PyPI. In order to prepare the necessary build environment the following packages have to be installed:

aptitude install python2.6 python-setuptools

The next line downloads and builds FlexGet itself:

easy_install flexget

The following step is optional. In case an interface to transmission-daemon is of use, the according FlexGet-plugin is installed like this:

easy_install transmissionrpc

Installation of Mailman in Ubuntu

*[MTA]: Mail Transfer Agent

 

Mailman is a powerful mailing list manager. It has a decent web interface for users and admins and offers content filtering, archiving, and much more. This documentation is based on Mailman v.2.1.14, Ubuntu 11.04, and Exim v.4.7.4 as MTA.

Package installation

To install the necessary packages this line is used:

aptitude install mailman exim4-daemon-heavy

Initial config

Unfortunately, the next steps are not part of the process of package installation but have to be done manually. First, a site password has to be set:

mmsitepass

Now, the initial list has to be created:

newlist mailman

Exim setup

To give Exim awareness of an existing Mailman setup, two ways are possible. The first one is based on the /etc/aliases file where new lists need to be added by hand. The second way is more comfortable, yet needs some customisation on the Exim part.

Alias file based

The previous step of creating the initial Mailman list results in the following output:

To finish creating your mailing list, you must edit your /etc/aliases (or
equivalent) file by adding the following lines, and possibly running the
`newaliases' program:

## mailman mailing list
mailman:              "|/var/lib/mailman/mail/mailman post mailman"
mailman-admin:        "|/var/lib/mailman/mail/mailman admin mailman"
mailman-bounces:      "|/var/lib/mailman/mail/mailman bounces mailman"
mailman-confirm:      "|/var/lib/mailman/mail/mailman confirm mailman"
mailman-join:         "|/var/lib/mailman/mail/mailman join mailman"
mailman-leave:        "|/var/lib/mailman/mail/mailman leave mailman"
mailman-owner:        "|/var/lib/mailman/mail/mailman owner mailman"
mailman-request:      "|/var/lib/mailman/mail/mailman request mailman"
mailman-subscribe:    "|/var/lib/mailman/mail/mailman subscribe mailman"
mailman-unsubscribe:  "|/var/lib/mailman/mail/mailman unsubscribe mailman"

So, the lines after "## mailman mailing list" are to be copied and pasted at the end of the file /etc/aliases. This has to be done each time a new list is created. The changes are activated by the following line:

newaliases
Exim config file based

The second way of configuration makes Mailman speak to Exim directly. Therefore, a couple of changes are to be made to different sections of the Exim config file at /etc/exim4/exim4.config.template.

In the section "MAIN CONFIGURATION SETTINGS" these lines are to be added:

# ---------- MAILMAN --------------------------------------------------------
# Home dir for your Mailman installation -- aka Mailman's prefix
# directory.
MAILMAN_HOME=/var/lib/mailman
MAILMAN_WRAP=MAILMAN_HOME/mail/mailman

# User and group for Mailman, should match your --with-mail-gid
# switch to Mailman's configure script.
MAILMAN_USER=list
MAILMAN_GROUP=daemon
# ---------- /MAILMAN -------------------------------------------------------

In "ROUTERS CONFIGURATION" just below "begin routers" these line are new:

# ---------- MAILMAN --------------------------------------------------------
# Messages get sent out with envelope from "mailman-bounces@virtual_domain"
# But mailman doesn't put such addresses in the aliases. Recognise these here.
mailman_workaround:
  domains = +local_domains
  require_files = MAILMAN_HOME/lists/$local_part/config.pck
  driver = accept
  local_parts = mailman
  local_part_suffix_optional
  local_part_suffix = -bounces : -bounces+* : \
           -confirm+* : -join : -leave : \
           -subscribe : -unsubscribe : \
           -owner : -request : -admin
  transport = mailman_transport
  group = MAILMAN_GROUP

# Mailman lists
mailman_router:
  domains = +local_domains
  require_files = MAILMAN_HOME/lists/$local_part/config.pck
  driver = accept
  local_part_suffix_optional
  local_part_suffix = -bounces : -bounces+* : \
                      -confirm+* : -join : -leave : \
                      -subscribe : -unsubscribe : \
                      -owner : -request : -admin
  transport = mailman_transport
  group = MAILMAN_GROUP
# ---------- /MAILMAN -------------------------------------------------------

In section "TRANSPORTS CONFIGURATION" just after "begin transports" the next line need to be added:

# ---------- MAILMAN --------------------------------------------------------
mailman_transport:
  driver = pipe
  command = MAILMAN_WRAP \
            '${if def:local_part_suffix \
                  {${sg{$local_part_suffix}{-(\\w+)(\\+.*)?}{\$1}}} \
                  {post}}' \
            $local_part
  current_directory = MAILMAN_HOME
  home_directory = MAILMAN_HOME
  user = MAILMAN_USER
  group = MAILMAN_GROUP
  freeze_exec_fail = true
# ---------- /MAILMAN -------------------------------------------------------

The new config is processed and set up by this line:

update-exim4.conf && service exim4 restart

Apache config

Mailman comes with a nice web UI and even a fitting config file for Apache. In /etc/apache2/sites-available/default the next two line are to be inserted right before the end of the file:

Include /etc/mailman/apache.conf
RedirectMatch ^/mailman(\/*)$ http://hostname.yourdomain.tld/cgi-bin/mailman/listinfo

The first line lets Apache parse the Mailman configuration file. The second line redirects all requests for http://hostname.yourdomain.tld/mailman to the listinfo page of Mailman at http://hostname.yourdomain.tld/cgi-bin/mailman/listinfo.

The changes are activated like this:

service apache2 restart

Workaround for list archives-bug in Ubuntu

In recent releases of the Mailman package for Ubuntu, the privileges on the directories of list-archives are not set properly so that new posts are not archived at all. To fix this the following line will help:

chown -R list /var/lib/mailman/archives/private/

In case there are already posts which are not indexed, the archive can be reinitialized like this:

/var/lib/mailman/bin/unshunt

Quick setup of Jabber-Server (ejabberd) in Ubuntu

 

This posts documents the necessary steps to setup a running Jabber-server in Ubuntu. It uses ejabberd v2.1.5 and by the time of writing it is tested on Ubuntu 11.04.

Installation of packages

First the according Ubuntu packages are supposed to be installed:

aptitude install ejabberd

Customizing the setup

Next step is to customize two settings in /etc/ejabberd/ejabberd.cfg:

%% Admin user
{acl, admin, {user, "youruser", "yourdomain.tld"}}.

%% Hostname
{hosts, ["yourdomain.tld"]}. 

The rest should basically fly by itself.

Now an administrative user has to be added. Is is done on the command line with:

ejabberdctl register youruser yourdomain.tld yourpassword

After this, the server can be started by the following line:

service ejabberd start

If all went well, now the administrative backend should be good to go. It is accessibly with this URL:

http://yourjabberhostname.yourdomain.tld:5280/admin

Customizing the DNS entry

In order to get your server compliant with most of the clients, your domain is supposed to offer the following SRV-entries (example shows bind zonefile entries):

yourjabberhostname       IN A       IP_of_jabberhost
_xmpp-client._tcp        IN SRV     5 0 5222 yourjabberhostname.yourdomain.tld.
_xmpp-server._tcp        IN SRV     5 0 5269 yourjabberhostname.yourdomain.tld.
_jabber._tcp             IN SRV     5 0 5269 yourjabberhostname.yourdomain.tld.

Have fun with your individual jabber server!

Testing webserver-security with Nikto

A decent tool to test a webserver's security setup is Nikto. It performs comprehensive tests on any given webserver and outputs its vulnerabilities. Using Ubuntu, the tool is easily available. It can be installed like this:

aptitude install nikto

To perform an actual test, the basic syntax looks like this:

nikto -h www.serverofinterest.tld

Depending on the extend of the test and the vulnerability of the server in question, the output might take a while. Yet, it will be quite enlightening as to where a server's problem are.

Enabling ModSecurity protection in Apache2 on Ubuntu

 

This post documents how to add ModSecurity protection in Ubuntu's Apache. It has been tested on Ubuntu 11.04 but might work on earlier or latter issues with few modifications.

Installation of base packages

ModSecurity is a open source Web Application Firewall (WAF). It offers an array of request filtering and other security features to the Apache HTTP Server.

Fortunately, ModSecurity is already part of the Ubuntu package repositories. Therefore, installing the necessary stuff is rather straightforward:

apt-get install libapache2-mod-security mod-security-common
a2enmod mod-security
/etc/init.d/apache2 force-reload

Activation of the core rule set

The mod-security-common-package contains the core rule set (crs) which will not be configured automatically. In order to activate the crs, /etc/apache2/conf.d/mod-security.conf has to be created with the following content:


    Include /usr/share/doc/mod-security-common/examples/rules/*conf
    Include /usr/share/doc/mod-security-common/examples/rules/base_rules/*conf

Now the web server has to be restarted by the following line:

service apache2 restart

Testing the setup

In order to prove the setup is working, a test file called test.php with the following content can be used:

It's supposed to be placed in the root of your web server, so that it can be accessed by http://yourserver.tld/test.php. To run the actual test, the following address will do:

http://yourserver.tld/test.php?secret_file=/etc/passwd

If the content of /etc/passwd is displayed, ModSecurity is not working. A working installation will show a "403 Forbidden" error message.

Hardening PHP on Ubuntu with Suhosin patches

This post documents how to enable the Suhosin protection system for PHP on Ubuntu installations. It has been tested on Ubuntu 11.04 but might work on earlier or latter issues with few modifications.

Apparently, Ubuntu repositories already contain the patches as an additional package. They are installed by the following line:

aptitude install php5-suhosin

After that, the patches are available but not yet configured. The central config file for the protection system is located at /etc/php5/conf.d/suhosin.ini. For a basic configuration uncomment the following lines:

[suhosin]
suhosin.executor.include.max_traversal = 4
suhosin.executor.disable_emodifier = on
suhosin.mail.protect = 2

; Filtering Options   
suhosin.cookie.max_vars = 2048
suhosin.get.max_array_index_length = 256
suhosin.post.max_array_index_length = 256
suhosin.post.max_totalname_length = 8192
suhosin.post.max_vars = 2048
suhosin.request.max_totalname_length = 8192
suhosin.request.max_varname_length = 256

Defining the latter options keeps phpMyAdmin working. After editing the file, the changes are activated by restarting the Apache daemon:

service apache2 restart

Now you may play around with the other config options in order to harden your webserver even more.

Full restore from Time Capsule with account-authentication

 

This week the people at Apple released their new OS X-Version called "Lion" (10.7). Some interesting new highlights in terms of UI, backup and encryption were announced. I was curious and gave it a shot.

To be on the safe side I did a full backup with Apple's rsync-based Time Machine-software on my Time Capsule backup device. This way, I expected to be able to do a full system restore of a pre-upgrade-state in case something goes south.

Mission Control unfortunately is a pain

After trying Lion for a very short while, I discovered that quite many of my favorite tools wouldn't work as they did in Snow Leopard. Examples are Quicksilver, MacBiff, PathFinder and GeekTool. I suppose that most of these tools will get an update sooner than later so I would've been able to live with that.

Yet, one thing really is such a bummer that I'm not going to use Lion in the near future: Mission Control basically killed Spaces, Exposé and the handy "show all windows" functions. I rely on these in my daily work by using "hot corners" and I don't want to be forced or patronized to change my well working processes. I guess I'm not the only nerd to consider this a step backwards, so I'm counting on some witty developers to revive projects like Desktop Manager or Virtue in order to rebuild the Snow Leopard-ish window management behavior.

Starting the restore

Long story short I now wanted to do a full restore of my Snow Leopard setup. Considering myself security concerned, I gave separate accounts on my Time Capsule to all machines which do a backup there. Then I booted my Snow Leopard-DVD and chose the option to do a full restore from a Time Machine-backup.

The restore helper program displayed the according backup and even accepted my auth-data. Yet, I couldn't start the backup. I tried the same from a different Mac and another backup: same issue.

I figured it must have something to do with the user accounts on the Time Capsule as I was able to restore a Mac back in the days when I had only one machine. I changed the file sharing-setting in AirPort Utility so that the Backups were just secured "With Time Capsule password". Having a look at the directory structure on the Time Capsule, I saw that the backups are not stored in the actual root directory. Due to using user accounts, there was now a structure of /Users/machineName.

Bug found and worked around

I suspected there might be a bug in the restore software, expecting backups always in the root directory of the Time Capsule drive. I moved the according sparse image of my machine to the root of the Time Capsule and tried the restore again. It worked!

The result is that the "Restore System from Backup" functionality on the DVD can't handle separate user accounts on a Time Capsule. Apple, you should be embarrassed about that, especially for not documenting and not fixing this.

Anyway, after the successful restore I moved back the sparse file and reactivated the Time Capsule-accounts to keep my security conscience calm. After that, all backups are working as they did before.

GPG secured Backups

 

There are some situations in which it becomes useful to store important and crucial data in non-trustworthy environments. One example can be an off-site backup at an external hoster where full control over the infrastructure can not be maintained. Then, encrypting your files comes in handy so that in an emergency the data can be restored and only the according key has to stay save.

Key creation

A new public/private key pair is generated by running

gpg --gen-key

Just follow the prompts. Depending on the use case and level of later automation it can be useful to skip the definition of a passphrase. Keep the key ID in mind for the next step (you can display it again with gpg --list-keys). The keys are now stored in the user's home directory in ~/.gnupg/secring.gpg and ~/.gnupg/pubring.gpg. Remember to keep these files save.

Encryption of data

Now, tar is used to archive directories and files and pipe the resulting package into gpg:

tar -c {dirs_and_files_to_backup} | gpg -r {gpg_key_id} -e -o backup.tar.gpg

The resulting backup.tar.gpg can savely be stored at some non-confidential place. If your data can be compressed, it may be useful to replace tar -c by tar -cz in order to compress the data during archiving.

Decryption of data

To retrieve your data from the encrypted archives, the previous process is basically reversed:

gpg -d backup.tar.gpg | tar xf -

If you compressed your data in the encryption process you need to replace tar xf by tar xzf.

Fresh install of OpenWrt Backfire from Mac OS X 10.6 by tftp

 

This article documents flashing OpenWrt Backfire on Linksys WRT54g v2 and on Asus WL-500g Deluxe and Premium. It uses OS X's built-in tftp to transmit the firmware to the router while it's in recovery mode. Recovery mode is active only for a few seconds in the beginning of the router's boot process.

Firmware retrieval

As of today, the most recent and solid firmware version is Backfire 10.03.1-rc4. The according download links are the following:

Network configuration of OS X

Disregarding the router's network setup before the following installation process, during rescue mode it will always fall back to 192.168.1.1/24. Therefore, if you do not already use a 192.168.1.0/24 subnet, you will have to to add a virtual interface to your Mac.

Therefore, in "System Preferences" open "Network" and follow these steps:

  • Click the little "plus"-sign in the lower left corner
  • Choose "Ethernet" and some random name for the interface
  • Click "Create"
  • Then, from the "Configure IPv4"-dropdown, choose "manually" as network setup
  • Enter "192.168.1.2" as "IP Address" and "255.255.255.0" as "Subnet Mask"
  • Confirm with "Apply" in the lower right corner

Now, you have an addition virtual interface on your shiny OS X-system by which you can feed the router with the new firmware.

Transmitting the firmware

Luckily, OS X comes with the tftp command, so there is no further setup on that part. Simply open a terminal window and type (or copy'n paste):

tftp 192.168.1.1
binary
rexmt 1
timeout 60
trace
Flashing the Linksys WRT54g

Now, type the following line but do not execute yet:

put 

Just before hitting "Enter", unplug and replug the router's power cable. After only a few seconds you should get a lot of output like this:

[...]
sent DATA 
received ACK 
[...]
Flashing the Asus WL-500g Deluxe/Premium

With the Asus routers, the process is basically the same. The difference is that while plugging in the router's power, the reset button has to be pressed. Also important to notice is that Asus routers keep their previous network setup in recovery mode, i.e. you don't have to adjust your network setup to 192.168.1.1. Yet, after flashing, the Asus devices will turn to 192.168.1.1.

First contact with the new system

The above output confirms the successful transmission of your firmware. If nothing went wrong, you will be able to reach your router by web access on http://192.168.1.1. Also, you can already use telnet to gain shell access:

telnet 192.168.1.1

Beware that in case you upgrade from a previous OpenWrt setup, the router might have recovered some of the old setting and therefore starts with the previous networking setup.

Now, you may follow the hints on http://wiki.openwrt.org/doc/howto/basic.config about basic configuration immedately after flashing OpenWrt.