Saturday 6 July 2013

Get-iplayer: The solution to the ERROR: RTMP_ReadPacket problem

If you have installed (or reinstalled) get-iplayer and it not longer works, but throw something like this:

Connecting ...
INFO: Connected...
ERROR: RTMP_ReadPacket, failed to read RTMP packet header
INFO: Command exit code 1 (raw code = 256)
WARNING: Failed to stream file [some program file].partial.mp4.flv via RTMP
INFO: skipping flashstd2 mode
ERROR: Failed to record '[the program name]'
The solution is to run this command:
get_iplayer --prefs-add --rtmp-tv-opts="--swfVfy=http://www.bbc.co.uk/emp/releases/iplayer/revisions/617463_618125_4/617463_618125_4_emp.swf"
If that doesn't work, make sure that you have lame, ffmpeg, RTMPdump and flvstreamer installed.

Saturday 15 June 2013

Temporarily disabling the touchpad when typing

Tired of typing and accidentally changing the cursor position? A little known trick in KDE enables you to disable the touchpad. It comes back after a configured amount of time.

1) Install kde-config-touchpad
2) K -> Applications -> Utilities -> Touchpad Management

Now you are able to configure the touchpad. You can select "Automatically turn off touchpad on keyboard activity".

This means that whenever you are typing, touching the touchpad will no longer cause you problems. After a short delay, once you have stopped typing, the touchpad will return into service.

Wednesday 30 January 2013

Thunderbird, Apache, Wevdav and Filelink: Rolling your own Thunderbird filelink

Thunderbird, Apache, Filelink

If you are sending an email with a large file attachment, the chances are that it will bounce back with a message saying that the email exceeds the maximum allowed.

Emailing large documents, presentations, images and other multimedia content is the norm these days. This means that bounced emails are a regular plague.

Thunderbird has solved this problem by implementing "Filelink" technology. This works by uploading the attachment to a server and converting the attachment into an embedded link in the email.

Several services are presently included in the Thunderbird offering, including Ubuntu One, YouSendIt and Box. The Mozilla page for filelink is here: Filelink.

However, most users are unlikely to want to trust their attachments to a third party service. There is a way to host your own solution, using Apache and turning on WebDav. 

First, you need to install the Thunderbird addon, Webdav for filelink.
  1. Go to Webdav for filelink and install the add-on in Thunderbird (alternatively, Tools->Addons in Thunderbird and search for it in there and install it).
  2. After you have configured your webdav service in Apache, you will need to configure Webdav for filelink so that it knows which server to log in to and how to authenticate.
Now you need to set up your Apache server so that it supports webdav. Of course you are running Linux.

These days, Apache on Linux usually has the directory /etc/httpd/conf.d. So in this directory, create the file webdav.conf with this content:
 
    LimitXMLRequestBody 50000000

    Alias /webdav "/home/httpd/webdav"
   
        Dav On
        Options +Indexes
        IndexOptions FancyIndexing
        AddDefaultCharset UTF-8
        AuthType Basic
        AuthName "WebDAV Server"
        AuthUserFile /etc/httpd/webdav.users.pwd
        Require valid-user
        Order allow,deny
        Allow from all
   

 
Otherwise, if you don't have the directory, then put that content into your httpd.conf file. httpd.conf is often found in the conf directory.

Note that you may choose to have your webdav directory something other than our setting (/home/httpd/webdav). You will need to create this directory:
  1. mkdir /home/httpd/webdav
  2. chown apache /home/httpd/webdav
  3. chgrp apache /home/httpd/webdav
  4. chmod 700 /home/httpd/webdav
for apache use the user and group name for your apache process.

Now you need to create the password file for authentication to the webdav service:
  1. htpasswd -c /etc/httpd/webdav.users.pwd myuser
  2. Repeat (1) for all the users that you which to grant access to, without the "-c" flag.
where myuser is the user you are giving access to the service to, probably the first time, your own username.

Open your httpd.conf. Check that it has these items in it:
  • LoadModule dav_module modules/mod_dav.so
  • LoadModule dav_fs_module modules/mod_dav_fs.so

  •     # Location of the WebDAV lock database.
        DAVLockDB /var/lib/dav/lockdb
Now run:
  • /etc/init.d/httpd configtest
  • if all is okay:  /etc/init.d/httpd restart
You are ready to test your webdav. In firefox, go to:
 http://mydomain.com/webdav
 (of course you substitute your own domain name for "mydomain.com"!).

You should be prompted to for a username and password, enter the credentials that you created above (in the htpasswd bit).

If that works, then you are nearly there:
  1. Open up Thunderbird
  2. Edit->Preferences->Attachments->Add
  3. Choose "Webdav"
  4. In location enter: http://mydomain.com/webdav
  5. Enter your authentication and choose "remember password" when prompted.
  6. Click OK and you're done
Thunderbird checks that it can authenticate on the webdav service immediatley. The actual verification should be almost immediate. There are two possible points of failure in the above:
  1. You entered the wrong username/password or the server entry isn't correct. This will generate authentication or server error messages in Thunderbird.
  2. The URL is recognised by Apache but it isn't the proper webdav URL. When this happens, Thunderbird tries and tries again for a long time, ultimately failing to authenticate.

From now on, when you create an email and put in one of more large attachments, Thunderbird will ask you if you wish to convert them into a link instead. You can also right-click on an attachment, choose "covert to" and then choose link.

NOTE: If, when you are composing your first email using a large attachment, your Thunderbird filelink keeps failing with "Webdav failed authentication", the chances are that your permissions are not correct for apache to write to the webdav directory on the server that you specified. Check your permissions! Thunderbird has already verified the logging in, so this is actually the wrong message being reported. Unfortunately, you probably won't see these permissions failures in the log files on the server either.

Thursday 17 January 2013

Oracle to Postgres conversion tool

 Do you have Oracle databases that you wish to convert to Postgres? A new version of the Oracle to Postgres conversion too, Ora2Pg, has been released.

About Ora2Pg 
Ora2Pg is an easy and reliable tool to migrate from Oracle to PostgreSQL. It is developed since 2001 and can export most of the Oracle objects (table, view, tablespace, sequence, indexes, trigger, grant, function, procedure, package, partition, data, blob and external table). Ora2Pg works on any platform and is available under the GPL v3 licence. Docs, Download & Support at http://ora2pg.darold.net/

About Postgresql
 Postgresql (Postgres) is an enterprise class database system (DBMS) that is OpenSource, free to use and free of licence fees, that is released under the PostgreSQL License, which is an MIT-style license.

It compares very favourably to enterprise grade databases such as OracleIngres (actian) and DB2. It is an upgrade from mid-range database products and light weight / embedded data stores.

Postgres is actively developed and has a large, active online community with many companies (including Kieser.net) offering commercial Postgres support.

Thursday 16 September 2010

Converting PST files to Linux MBOX format

You want to upgrade from Outlook to Thunderbird and you've changed from Windows to Linux as a desktop. If you have control over a mail server running dovecot, you are in luck.

The tool that you need is the very handy libpst library, although if you run Kubuntu or Ubuntu then this is available on synaptic.

Simply export your Outlook mail into a pst file. Let's call the file mailBackup.pst. Now copy the file to your email server or linux desktop.

It is important to create a directory to work in because for each folder in the PST (Outlook) file, there will be a mbox file created with the same name. Cd into this directory and keep emailBackup.pst in the directory above it.

Now simply run:
   readpst ../emailBackup.pst

The Outlook file will be converted into a series of mbox files. If you run Dovecot, these can be put into your mail directory (be careful not to overwrite existing files!) and you will be able to access them with Thunderbird or your webmail software (I recommend horde).

You are also able to open mbox files in Thunderbird.

Thursday 3 June 2010

SpamAssassin (spamd) deadlocks, runnning slow, Postgres

The usual way to use spamassassin is to implement the Bayesian filtering database in flat files. That's the default configuration and most distros ship with this option.

But that's a bad option. If you want to implement a database, use a proper database.

The advantages are numerous: Proper caching and cache management, ability to tune the database and tables, better locking capabilities (as in the locking capabilities exist), clustering and fail-over capabilities which all amount to better resource usage, better throughput, less server strain, better maintenance and a much better service.

There is no better enterprise class OpenSource database than Postgres (or it's twin, Ingres). Frankly, your decision to NOT use Postgres for your database needs has to be very well justified as PG is not only truly enterprise class, but it's also easy to set up, easy to admin and, most importantly. you can tune it properly.

Which is why you should be using Postgres with SpamAssassin.

Unfortunately, though, the latest version (and previous versions), 3.3.1 have some pretty bad SQL in them. Rather than utilise the PG strengths and keys, the SQL has not been optimised from a performance point of view. Which, in an email system, is one of the most important things to consider!

The biggest hole is the use of the SQL IN operator on the bayes_token table. This effectively forces a full table scan because the unique key is id, token. On a system-wide implementation, the ID column is a particularly weak key (i.e. not a key at all because it's always the same value) so this is a real deal-breaker.

The solution is to use the primary key wherever possible, which, it turns out, is nearly all the time.

On a system with a large spam database, this is the difference between a powerful server grinding to its knees v.s. the same server flying at vast throughput.

The biggest deal-breaker is in the update of the atime column, which is about the most regularly performed task. So it's the hottest of the hot spots in the spamd PG code and also the worst implemented. The fix, however, is very easy.

Simply edit this file (note the path will be different on your machine:
/usr/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/BayesStore/PgSQL.pm

and make these changes:


Original code fragment:
-------------
sub tok_touch_all {
.
.
.
  my $sql = "UPDATE bayes_token SET atime = ? WHERE id = ? AND token IN (";

  my @bindings;
  foreach my $token (sort @{$tokens}) {
    $sql .= "?,";
    push(@bindings, $token);
  }
  chop($sql); # get rid of trailing ,

  $sql .= ") AND atime < ?";

--------------



Amendments:
----------

sub tok_touch_all {
.
.
.

  foreach my $token (sort @{$tokens}) {
  my $sql = "UPDATE bayes_token SET atime = ? WHERE id = ? AND token =";

  my @bindings;
    $sql .= "?,";
    push(@bindings, $token);
  chop($sql); # get rid of trailing ,

  $sql .= " AND atime < ?";

.
.
.
  $self->{_dbh}->commit();
  }

  return 1;
}

----------------------------

Note that I insert the closing } before the "return 1".

I.e. I have converted this into a line by line update, so that the DB can use the very strong primary key of id,token.

The performance difference that this makes is absolutely enormous on a busy system.

In case you want to simply cut and paste the entire function, here is the tok_touch_all function with the amendments in it:

sub tok_touch_all {
  my ($self, $tokens, $atime) = @_;

  return 0 unless (defined($self->{_dbh}));

  return 1 unless (scalar(@{$tokens}));

  foreach my $token (sort @{$tokens}) {
  my $sql = "UPDATE bayes_token SET atime = ? WHERE id = ? AND token =";

  my @bindings;
    $sql .= "?,";
    push(@bindings, $token);
  chop($sql); # get rid of trailing ,

  $sql .= " AND atime < ?";

  $self->{_dbh}->begin_work();

  my $sth = $self->{_dbh}->prepare_cached($sql);

  unless (defined($sth)) {
    dbg("bayes: tok_touch_all: SQL error: ".$self->{_dbh}->errstr());
    $self->{_dbh}->rollback();
    return 0;
  }

  my $bindcount = 1;

  $sth->bind_param($bindcount++, $atime);
  $sth->bind_param($bindcount++, $self->{_userid});

  foreach my $binding (@bindings) {
    $sth->bind_param($bindcount++, $binding, { pg_type => DBD::Pg::PG_BYTEA });
  }

  $sth->bind_param($bindcount, $atime);

  my $rc = $sth->execute();

  unless ($rc) {
    dbg("bayes: tok_touch_all: SQL error: ".$self->{_dbh}->errstr());
    $self->{_dbh}->rollback();
    return 0;
  }

  my $rows = $sth->rows;

  unless (defined($rows)) {
    dbg("bayes: tok_touch_all: SQL error: ".$self->{_dbh}->errstr());
    $self->{_dbh}->rollback();
    return 0;
  }

  # if we didn't update a row then no need to update newest_token_age
  if ($rows eq '0E0') {
    $self->{_dbh}->commit();
    return 1;
  }

  # need to check newest_token_age
  # no need to check oldest_token_age since we would only update if the
  # atime was newer than what is in the database
  $sql = "UPDATE bayes_vars
             SET newest_token_age = ?
           WHERE id = ?
             AND newest_token_age < ?";

  $rows = $self->{_dbh}->do($sql, undef, $atime, $self->{_userid}, $atime);

  unless (defined($rows)) {
    dbg("bayes: tok_touch_all: SQL error: ".$self->{_dbh}->errstr());
    $self->{_dbh}->rollback();
    return 0;
  }

  $self->{_dbh}->commit();
  }

  return 1;
}

Monday 29 March 2010

No sound in flash on AMD 64 Kubuntu

The standard ubuntu/kubuntu installation breaks sound in flash. There are several reasons for this, the first that Kubuntu/Ubuntu uses pulse audio and flash (along with many other applications) doesn't play nicely with pulseaudio. However, PulseAudio problems is just the first of two issues. The second is that Adobe 32 bit flash cannot share audio devices and output sound in the shared environment. For that you need the 64 bit version.

So the first port of call in Kubuntu is to go to system settings/multimedia/device preference/audio output/Music and bump PulseAudio to the preferred (top) device.

Click on "test" to make sure that pulseaudio is actually working on your system. If you don't have a sound coming out, you need to first get pulseaudio working properly.

Next, you need to download the 64bit version of flash from Adobe and you can do this here:
http://labs.adobe.com/downloads/flashplayer10_64bit.html

(note the above is for version 10, you may wish to check that it's still the latest 64bit version).

Unzip the file. It gives you a libflashplayer.so file.

Now you need to overwrite this for firefox (and other applications that use flash).

For example in my system, in my user home directory, I have ~/.mozilla/plugins/
This is the directory that firefox/mozilla/chrome loads flash from. So I copy the file into this directory, overwriting the existing 32 bit version.

You may want to

sudo find / -name libflashplayer.so -print

to find out where this file is elsewhere on your system and update it there too.