As usual, I have used an address I can blacklist or destroy if needed (catchall rules!), just to try to find out who steals or buys addresses from others.

It seems the company Computer Futures, is one of the companies that actually steals e-mailaddresses from different sites. I checked with CodeQuest.nl, they have not given or sold my address to Computer Futures, but still, my address popped up in their mailinglist for some reason.

So, not only doesn’t Computer Futures know what kind of people they want to hire to your company (have some experience with this while working for a client of mine, they forged the résumé of this particular person by adding stuff and making stuff up), they also steal e-mailaddresses for no apparent reason.

I do wonder why people want to do business with such a company.

Some of the content of the mail I received:

We are sending you this message because you are registered on the Computer Futures system.  At computer Futures we are committed to protecting the privacy of our users.

Wir schicken Ihnen die folgende Mitteilung, da Sie auf dem System der Computer Futures Solutions registriert sind.  Computer Futures Solutions setzt sich dafür ein den Datenschutz seiner Benutzer zu wahren.

Ce message vous est envoyé car vous êtes enregistré sur le registre de Computer Futures.  Computer Futures est commis à protéger les données personnelles de nos utilisateurs.

Wij sturen u dit bericht omdat u geregistreerd bent in het systeem van Computer Futures.  De bescherming van de privacy van onze gebruikers staat hoog in het vaandel bij Computer Futures.

Data Protection Act Fair Collection Notice: Computer Futures Solutions Ltd will process your data at their offices in London.  We will record information relevant to the provision of IT recruitment services to your company, including your name, business address, telephone number, e-mail address and details of your requirements.  We will use the data to forward to you information on candidates/IT consultants and projects and services we think maybe of interest to you. We may pass your information on to other parts of our group who provide related services.  We will hold your data as we may be able to provide you with useful information and services in the future.  If you do not want to receive such communications, wish to update your registration or have any queries concerning this statement, please contact  <mailto:data-audit@compfutures.com> data-audit@compfutures.com.  To view our Privacy Statement please click here …http://www.computerfutures.co.uk/stateviewer.php?file=CF%20UK%20(English)%20Client.txt

…

Computer Futures Solutions is on of Europe’s largest IT staffing business, with a network of offices throughout the UK and Europe.  We have an unparalleled ability to recruit on either a contract or permanent basis, all types and level of IT professional.

By investing extensively in technology we provide you, our client, competitive advantage in the “War for Talent”.  This investment will continue, keeping Computer Futures Solutions at the forefront of the staffing industry.

…

Use Discovery, our sophisticated search facility, to access one of the
most comprehensive databases of contractors in Europe. <– I just get the idea, they harvested this database from other sites
http://www.computerfutures.com/index.php?dir=olr&sec=employers

Okay, I admit, I’m a bad boy, as I’m not following the ‘the mail is strictly confidential’ part, but since this was SPAM anyways and, well, I did not agree to the following:

This electronic transmission is strictly confidential and intended solely
for the addressee. If you are not the intended addressee, or have
otherwise received this transmission in error, you must not disclose,
copy or take any action in reliance of this transmission.

I hope any reader of this message will think twice before doing business with these spammers.

And yeah, I validated this actually originated from their servers. At least, it was the same network supplier, with the IP’s registered to the same address in the RIPE systems…

Please note, I’m saying this all in the form of personal opinion and free speech.

If any employee of Computer Futures would like to give me a valid explanation as to why an e-mail address, solely in existence for one site (which is not affiliated with Computer Futures), is spamming me, please contact me .

And yeah, I know this is badly written, I’m just pissed off…

Why does this always happen when you don’t have the time for it. Receiving backscatter bombs.

One of the big issues, I still have to deal with is what to do about the endless stream of backscatter bounce messages.

First of all… What is backscatter, you might ask.

What is backscatter?

Backscatter is the effect you get when you have a mail server, that starts to receive bounces of mails you didn’t send. Like the following example:

A spammer decides to send a spam to all kinds of addresses all over the world, with your e-mail address in the From. And those messages are bounced by the receiver.

Normally, what would happen is, the sender would get a nice MTA generated e-mail stating the mail could not be delivered. This is exactly the problem. Instead of the sender (the spammer), the mail gets send to you, due to the fact your domain was abused… This is hard to filter, as the MTA’s that are sending the messages to you, are actually following all the relevant RFC’s. So it’s hard to figure out what to accept and what to deny. But it gets even worse, when you have a fallback mailserver .

You can’t just refuse everything that doesn’t feel or sound like something you should accept. For example, in a new setup I’m working on, I required all mail that had a from which matched a local domain, to be authenticated, works nice in theory (although this doesn’t solve this issue), however… If you have someone that uses the SMTP server of his or hers provider and cc the messages to you, your local MTA will just refuse the mail. Which is bad…

To make it even more complicated, some people use a secundairy or fallback mailserver, you can just use greylisting, stuff like Sanesecurity addons for ClamAV, but you will still get more than enough crap mail to handle.

One thing that will help against backscatter in Exim, is a nice and ugly (RFC breaking) solution. You add something like this to your config:

BOUNCE_ID = <some kine of unique ID>
BOUNCE_SECRET = <some kind of md5 or sha1 key>

And later on, in your config, you add something like this to your ACL:

deny  senders = :
! condition = ${if match \
{$message_body $message_body_end} \
{[xX]-bounce-key:\\s*BOUNCE_ID;${rxquote:${lc:$recipients}};(\\d+);(\\w+)} \
{${if eq {$2} \
{${length_8:${md5:BOUNCE_ID;${lc:$recipients};$1;BOUNCE_SECRET}}} \
{${if <{${sg{${eval:$tod_epoch-$1}}{-}{}}}{864000}{1}}}}}}
message = Bounce does not contain a valid X-bounce-key signature so not accepting message

And to your remote smtp router, you add:

headers_add = ${if eq{$return_path}{}{}{X-bounce-key: BOUNCE_ID;${lc:$return_path};$tod_epoch;${length_8:${md5:BOUNCE_ID;${lc:$return_path};$tod_epoch;BOUNCE_SECRET}};}}

This will probably work really well, but it will also block certain stuff you do want to receive.

Like reading confirmations from Outlook.

To avoid that, you can add something to your ACL’s like:

accept
regex = [dD]isposition.*disposition-notification.*
accept
regex = [dD]isposition.*MDN-sent.*displayed.*

But this can also be used by spammers .

So far I have to admit, I disabled the ‘solution’ again, and just abuse SpamAssassin as much as possible, together with ClamAV / Sanesecurity and RBL checking.

If you don’t accept the message, it’s not your problem .

And still, you will receive loads and loads of spam, from the secundairy mailserver. Greylisting, SA on SMTP time, ClamAV on SMTP time is just not enough…

However, if you combine it with stuff like http://wiki.apache.org/spamassassin/WrongMXPlugin, it might be enough.

If anyone has better idea’s (that don’t involve shutting down the MTA entirely, I’m all ears!)

Dus, iedereen die zich afvraagt waar ik mee bezig ben, dit is dus een voorbeeld van een systeem waarmee ik bezig ben .

Het originele bericht;

Avetica introduceert de module Moodle Payments. Met deze gratis module kunnen betaalde cursussen aangeboden worden met Moodle, een soort webshop voor cursussen. De eerste betalingsmogelijkheid is met betalen via iDeal.

De Moodle-site eigenaar kan op deze manier zich volledig concentreren op de inhoud van de cursus, want betalen en factureren gaat volledig automatisch. Ook hoeft de Moodle-site eigenaar geen financiële investeringen te doen en ook geen iDeal-abonnement af te sluiten. Met de wekelijkse uitbetaling krijgt de Moodle-site eigenaar het geld op zijn/haar bankrekening gestort. Rapportages over het aantal afgenomen cursussen en alle uitgekeerde bedragen worden per e-mail verstuurd.

Met Moodle Payments heeft Avetica een nieuwe dienst toegevoegd aan de categorie e-learning & marketing.

Wel zorgde dit ervoor dat er ruim voldoende tijd over was om eens te gaan kijken naar alle stands op de expo.
Hier stonden behoorlijk wat interessante dingen bij, maar jammer genoeg meer gericht op grotere bedrijven, dan op MKB.

Novell & SuSe
Zucht…
Een aantal lezingen waren georganiseerd door Novell. Op zich waren de onderwerpen erg interessant, maar jammer genoeg meer gericht op het genereren van verkopen, dan op het verstrekken van informatie.
Desalniettemin zijn er toch behoorlijk wat interessante dingen besproken, al vraag ik me bij sommige punten toch af of het niet te duidelijk was dat het verkooppraatjes waren.

Norman
Een erg interessante lezing was de lezing van Norman, over malware detectie en sandboxes om uit te zoeken wat bepaalde malware precies doet.
Dit blijkt veelal gebruikt te worden binnen forensische wetenschap, om er zo achter te kunnen komen waar malware vandaan komt, wie het gemaakt heeft, en wie het bestuurd.
Hierbij werd uiteindelijk nog een mooi voorbeeld getoond, een virus (de naam is me even ontschoten), wat de sandbox liet connecten naar een IRC server, en daar stond te wachten op instructies voor aanvallen of spamruns.
Spijtig genoeg is deze software nog niet voor Linux beschikbaar, dus was het een beetje een vreemde eend in de bijt op LinuxWorld, maar wel een zeer interessante.

MythTV
De laatste lezing die ik op mijn agenda had staan, was er eentje over de implementatie van MythTV als mediacenter. Binnen de lezing werd besproken wat voor mogelijkheden er precies zijn in de laatste versie van MythTV, en werd er besproken wat voor hardware het handigste zou zijn.
Zelf heb ik al geruime tijd een MythTV mediacenter, en ben aan de hand van deze lezing toch nog even gaan knutselen.
Inmiddels heb ik de Mythstream plugin volledig werkend, wat er nu voor zorgt dat het ook mogelijk is om on demand dingen te bekijken, denk hierbij aan uitzending gemist, of de Shoutcast directory.

Al met al was het een zeer interessante dag, en waarschijnlijk toch iets te kort. Het blijkt later dat dit toch het type expo / conferentie is waar je eigenlijk 2 dagen naartoe zou moeten gaan.

Well, my article (here) doesn’t appear to be the only article about Dreamhost.

Just taking a look at stuff like this is almost fun to see (note this customer is on a totally different server): http://elliottback.com/wp/archives/2007/05/03/dreamhost-sucks-at-hosting/.

Or other stories, at:

• http://www.upstartblogger.com/why-dreamhost-sucks
• http://gerard.wordpress.com/2006/01/07/dreamhost-hosting-sucks-big-time/
• http://scobleizer.com/2006/09/28/dreamhost-getting-sucky-pr-out-on-blogs/
• http://www.azeemazeez.com/blogs/boo-dreamhost/
• http://www.thelastminuteblog.com/2006/07/27/dreamhost-is-really-starting-to-suck/
• http://www.rc3.org/2005/12/trash_dreamhost_here.php

I really hope these kind of messages gets picked up by the community some time, to force Dreamhost to get their act together!

Note, I tried some of the same kind of commands to prove it’s just policy to place too many users on their servers:

willie:~$ wc -l /etc/passwd
921 /etc/passwd

Meaning there’s a stunning amount of 921 users on the server. Will, let’s just say there’s about 20 to 30 users needed for normal system operation, it still leaves over 800 users!! – Note, I usually won’t place more than 200 to 300 domains (that’s about 75 users) on a server.

Furthermore, this is something I know as well;

Dreamhost and I have been having conversations now for a while about a site which gets 1-2k visitors, and hosts 51GB of transferred static content a day. I thought you might be interested in reading them. On 4/30/2007, I received this email from Brian S. about my site:

Connections to your domain ( static.imgfly.com ) crashed the shared apache service several times this morning. A connection limit has been placed on your site. Being on a shared server means you need to share the resources with other customers. Due to the heavy volume of traffic, other domains on the same service were not able to load. Once the traffic to your site has taperd off, we will gladly remove the connection limit. Please read the appropriate section of our Terms of Service and let us know if you have any questions.

dreamhost.com/tos.html

Okay, granted, I use a ‘little bit’ more traffic (I use about 1 TB per month), but still, the same issues!

People, please realize, Dreamhost is cheap. Cheap doesn’t mean good!

I ordered the ‘Crazy domain insane’ package, with 200 GB space, and 2 TB traffic. Which was supposed to be for $190.80 (for 2 years), but due to a coupon code, I was able to get this reduced with another $70 or something.

It sounded too good to be true. Well it was (and is).

Is it really crap?
First of all, they provide SSH access, nice and all, but you can clearly see the domain names of other customers, so you aren’t chrooted.
Note that I didn’t continue to ‘penetrate’ their security much further, as I don’t want to be suspended.

But allowing all users to work in a non-chrooted environment with ssh access is really the best way to go as a budget webhosting provider! (NOT)
Secondly, I know when someone logs in, as the ‘last’ command displays all output (ftp / ssh) for opened sessions.
Furthermore, the load is always above 3.something for the server I’m on. Even when it’s freakin’ midnight in the USA.

07:49:58 up 2 days, 21:24, 3 users, load average: 6.77, 6.80, 7.29

Oh and yeah, I can see the IP addresses of others with the ‘last’ command.

Benchmarking budget webhosting
Well, some time after I got the account, a friend asked me if I could sponsor him a few 100 GB’s of traffic on my own company’s hosting platform. Well, after thinking about it, it was a nice test for this account. So after some time, he linked his site nfshome.com to have all the demo’s retrieved from my site (icheb.nl).
After a while, everyone started getting 503 messages from Apache (‘service temporarily unavailable’), so I mailed Dreamhost.com.
It appeared my site was maxing out their capabilities (at less than 1 TB of traffic per month).
So they increased the available resources to the site. (Thanks Javier )

But 2 months later, after trying to deploy some test script, I made a mistake.
I accidentally used the wrong php-cli binary, effectively using almost all the resources the server had to offer (side effect of that program with PHP 4). So my account got suspended with the following message:

Hello,

We have turned off your website icheb.nl due to resourse usage, your user
account is at 1934.05cp when an avaerage user is around .1 thats less
then one, your using 49.00% of the whole server, this is not acceptible.
I see that most of the traffic is going to

GET /mirrors/nfshome/nfsmwdemo.exe

Remove this file as you are not permitted to offer the Need for Speed:
Most Wanted demo on our server to distribute. These servers are note made
for this, if you wanted to do this you would have to get a dedicated
server

You will need to contact us to let us know that you have removed it and
all other files like it before we will turn the site back on.

Thanks!

Javier

So, I guess hosting a few demo’s of games requires an dedicated server. Even when you’re buying enough bandwidth.
My reply was as follows:

Hello Javier,

I’ve taken the PHP-CLI script possibly causing the problem offline.
The PHP-CLI script in question requires more uptime to function correctly, it failed horribly with the power outage downtime / maintenance.

The nfsmwdemo.exe file isn’t something I’d like to take offline, as this is *not* using PHP.
Furthermore, the Dreamhost TOS doesn’t say anything about not being allowed to mirror / host demo’s.
I mostly hate this because the site was taken offline without contacting me first, which at this time is causing problems for some open source projects as well.

Dreamhost offers 2352 GB of traffic at this time, on a monthly basis, which is something, at least the first 2000 GB, I paid for. At least, that was the general intention.
Furthermore, in the last few days most stuff on the server is very slow, and the load seems a bit high. A load avarage of ‘load average: 11.72, 11.66, 13.46′ is quite about what a server with just two processors should have…

So, to clarify, I do not want to remove the .exe, however, the php-cli application causing the problem is no longer online. I actually do know what caused the problem. It tends to use a lot of cpu when started in php 4, but works nicely in php 5…

Wanting to sell me a dedicated server, just to host something taking less than 60% of the traffic limit, is just insane; how are your customers ever going to even reach the traffic limit, if that’s made impossible?
Just to clarify a few points; I own a webhosting company in my country, and am just using Dreamhosts services for some personal stuff… So I know what I’m talking about. A dedicated server based on the amount of traffic caused by one file is totally insane…

I hope we can get this problem out of the world, as the php-cli script causing the problem has been ‘disabled’, so to say.

With regards,
Icheb

Half a day later, the account was back online and I even got apologies (“I’m very sorry for the confusion, but as long as that file isn’t copyrighted then you’re free to host it with us if you need to.”), furthermore ‘Mike S’ agreed with me the load was a bit highish, and they would take care of it. Which later resulted in a bit more normal load, or at least something the server could technically handle.

Two months later, there was another problem, this time Apache wasn’t ‘balanced’ anymore, so it could not handle the requests. Whatever they did helped, and my site was available again.

But today, I received:

Hello,

I had to place a throttle on your domain icheb.nl due to the very large
number of hits it was recieving, effectively crashing the webserver.

James

So, what I think about it?
Well, > 70% of the requests to icheb.nl now receive a nice 503 error from Apache (it took me more than 20 tries to get into my own freakin’ site stats!!). Nice and all, but ehm, I’m still below 1.5 TB of the 2.2TB I’m able to use.

So what’s the big problem?
Well, Dreamhosts effectively seems to disable sites that do more traffic than they want you to use. So offering 2 TB traffic (that 2000 GB traffic) is nice, but they are limiting you before you even get close to it.
The Need for Speed demo’s I host are quite large, so there only is disk activity and Apache activity, the mirror doesn’t use any PHP (yeah, it does do some php, but that’s handled by one of my own Dutch servers).

Furthermore, the speed is limited. Download speed to The Netherlands seems to be limited to some random value around 50 to 400 KB/sec. While writing this, I’ve enabled a download (www.icheb.nl/100mb.bin which is 100 mb of random data) to a dutch server (with 100 MBit connection to the Internet).
To prove something is wrong, I enabled two more downloads, one from another dutch datacenter & one from a school network I have a shell account on.
Below you can see the output of wget (Linux download tool), and the speed measured:
Dutch 100 Mbit connection:

–14:11:32– http://www.icheb.nl/100mb.bin
=> `/dev/null’
Resolving www.icheb.nl… 66.33.223.235
Connecting to www.icheb.nl[66.33.223.235]:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 104,857,600 [application/octet-stream]

100%[====================================================>] 104,857,600 67.18K/s ETA 00:00

14:35:26 (71.43 KB/s) – `/dev/null’ saved [104857600/104857600]

So, that’s 71 kilobytes per second, not very much eh?

Other Dutch datacenter:

Resolving www.icheb.nl… 66.33.223.235
Connecting to www.icheb.nl|66.33.223.235|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 104,857,600 (100M) [application/octet-stream]

100%[=======================================================>] 104,857,600 65.31K/s ETA 00:00

16:52:14 (86.99 KB/s) – `/dev/null’ saved [104857600/104857600]

W00t, a whopping 86.99 kilobytes per second.
(Note, this is almost FIVE times as slow as the download of my ADSL connection)

The school network (at least 10 mbit):

Resolving www.icheb.nl… 66.33.223.235
Connecting to www.icheb.nl[66.33.223.235]:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 104,857,600 [application/octet-stream]

100%[====================================>] 104,857,600 57.48K/s ETA 00:00l

16:55:53 (77.11 KB/s) – `/dev/null’ saved [104857600/104857600]

This is more than FIVE and a half times as slow as my own ADSL download (I have 4 mbit).

My end conclusion
I think I have successfully proved Dreamhost is unable to give you what you buy, if you’re using more than 50% of what you buy. So they’re overselling, and people are just falling for it!.
Furthermore, their on server security is not 100% of what it should be.
I should not be able to see the domains of other users, or even their usernames. Example:

~$ ls -alh /home/rcc7369.wtf.22875/
total 40K
drwxr-sr-x 4 root staff 4.0K Oct 3 2006 .
drwxrwsr-x 454 root staff 28K May 4 04:03 ..
drwxr-sr-x 3 root staff 4.0K Oct 3 2006 XXXXXXXXXXX (removed by me).com
drwxr-sr-x 2 root staff 4.0K Oct 3 2006 ninjaweasel

I should not be able to see stuff like this:

~$ locate mysql.php
/home/rcc7369.wtf.22875/.com/home/mambots/content/geshi/geshi/mysql.php
/home/robotstephe.wtf.22203/.com/mambots/content/geshi/geshi/mysql.php
/home/swoleary.wtf.18742/.com/forum/db/mysql.php
/home/timesup.wtf.1954/.com/mambots/content/geshi/geshi/mysql.php

Furthermore, I’m even able to SEE the contents of the files locate give back!!!!
So, I can actually see files from other users. Granted, it depends on the chmod of a file (global read has to be on).

And yeah, they’re still running Linux kernel 2.4:
~$ uname -a
Linux willie 2.4.32-grsec+f6b+gr217+nfs+a32+fuse23+tg+++opt+c8+gr2b-v6.194 #1 SMP Tue Jun 6 15:52:09 PDT 2006 i686 GNU/Linux

But it isn’t a real issue, I too still have a few servers running on 2.4 (custom) kernels .

So, what the heck are these guys doing exactly?!!
Is this the way shared hosting is supposed to be done?

Note: I am not writing this just to be negative, but hopefully to allow people to realize it’s not wise to just go for the most cheapass hosting account they can find anywhere.
I hope someone at Dreamhost will take this seriously, and try to fix the problems. (And no, I don’t want my account to be suspended, I was planning to post an Asterisk howto today, but the flash movie is hosted at icheb.nl, so I don’t want to that yet…)

Hello,

Unfortunately, we cannot allow one user to negatively effect the server
we offer other users. All bandwidth is not created equal. Even if you’re
not hitting your bandwidth limit, you can still cause problems.

We cannot allow you to effect other users, so if you plan on
maintaining this amount of traffic, you’ll need to move to a dedicated
server, as you would have outgrown shared hosting.

If you need anything else, please let us know.

Thanks!

Brian

Furthermore, please understand, I can’t just arrange it on another server, bandwidth in the Netherlands is too expensive for these mirrors at this time. Also, please don’t DDoS me for this… If you really want to DDoS anyone for not getting your downloads, please do that with dreamhost .

After writing some scripts, I’ve able to recover everything (I think)…

Wtf did I do?
Today, I was working on transferring old backups offsite to our office backup drive. I used to use a nice script for this I wrote back in 2004 called shifter.sh. It would create the correct dirs (by date) for each reseller on each server, and move the latest backups there, after that, it would chown everything back the way it should be to allow transport/control by ftp. Well, that’s what it should do.

Due to something called ‘old stuff that shouldn’t be used anymore‘ I recently moved all clients away from a certain server. So I opened up the shifter file and deleted the lines belonging to the /home/backupsys/horus/ dirs (or server). Well, took about 2 minutes. What I didn’t see though, was that the code for the next server had an ‘cd ..’ 2 times, to get back to /home/backupsys from the /home/backupsys/{date}/.

So far so good, I ran the script, however it seemed to be taking longer than I could remember. After reading some stuff on the daily wtf some time ago about moving stuff wrong, I canceled the script and tried to see what the heck went wrong. Well, first of all, the dirs I needed to continue weren’t there. Secondly, I found this desturbing image:

ls -alh /

drwxr-xr-x   23 root     root         4.0K Nov 18 14:19 .
drwxr-xr-x   23 root     root         4.0K Nov 18 14:19 ..
drwxr-xr-x    2 backupsys backupsys     4.0K Nov 18 14:19 11-18-06
-rw-------    1 backupsys backupsys      13K Nov 18 14:21 aquota.group
-rw-------    1 backupsys backupsys      15K Nov 18 14:21 aquota.user
-rw-r--r--    1 root     root            0 Oct 11 18:29 .autofsck
drwxr-xr-x    2 backupsys backupsys     4.0K Apr  2  2005 bin
drwxr-xr-x    3 backupsys backupsys     4.0K Aug 20  2004 boot
drwxr-xr-x   20 backupsys backupsys     116K Oct 11 18:30 dev
drwxr-xr-x   46 backupsys backupsys     4.0K Nov 18 00:10 etc
drwx--x--x   22 backupsys backupsys     4.0K Oct 27 22:21 home
drwxr-xr-x    2 backupsys backupsys     4.0K Jan 25  2003 initrd
drwxr-xr-x    9 backupsys backupsys     4.0K Jun 10  2005 lib
drwx------    2 backupsys backupsys      16K Aug 19  2004 lost+found
drwxr-xr-x    2 backupsys backupsys     4.0K Jan 28  2003 misc
drwxr-xr-x    2 backupsys backupsys     4.0K Aug 18  2004 mnt
drwxr-xr-x    2 backupsys backupsys     4.0K Jan 25  2003 opt
dr-xr-xr-x  101 backupsys backupsys        0 Oct 11 20:29 proc
drwxr-x---   13 backupsys backupsys     4.0K Nov 18 14:22 root
drwxr-xr-x    2 backupsys backupsys     8.0K Jun 10  2005 sbin
drwxrwxrwt   12 backupsys backupsys     4.0K Nov 18 14:23 tmp
drwxr-xr-x   18 backupsys backupsys     4.0K Aug  6 14:38 usr
drwxr-xr-x   19 root     root         4.0K Oct 21  2004 var

Wha does this mean?
What this means, besides the /var every dir is owned by the wrong user. The script was killed by me while it was processing the /usr/. I am happy about the fact I killed it, because the next step would be to cd .. a few more times, and then starting to move everything again. Which would have meant, it would have created a situation like described here.

So, WTF happend?!!
Well, rather easy, there were too much cd ..‘s in the shifter.sh code. So ultimately it went to the /. And due to the fact I usually am too lazy to use the right user for these processes, I had root. So the script started to chown EVERY file on the / recursively to ‘backupsys:backupsys’. This usually is very bad. Luckily I canceled the process before the shell died, but I was unable to open a new shell from ssh.

Repairing the problem
As I said, I’m lazy, so I needed a fast way to repair this. A friend of mine, while laughing his arse off, thought of a nice idea, the ugly type of idea’s I’m known of (at least, that’s what some people think of me ). I have another old server with the same distro, so I created a quick and dirty script in PHP to write a ‘chown script‘. This script created chown commands based on the server it is on, which can then be transferred to the server with the ‘owner problem’. After using this script, I used another script I created earlier for Direct Admin, to restore /home ownerships.

In total, it took about 2 hours to f*ck the server up, and restore it. And I did it without any noticeable downtime on ANY service .

Does this still qualify for The Daily WTF?

(second note; this post might look a bit f*cked. This is caused by something in the WordPress editor thingy, it won’t parse my entered crap right)