Main > SystemAdministration > ReportOnTwikiCompromise	TWiki webs: Main | Department | TWiki | Odysseus | Public | Sandbox
	Main . { Users | Groups | Changes | Index | Search | Go }

Report on our November 2004 twiki compromise

Background
----------

Acheloos is an i386 Debian 3.0 (woody) server providing several
services to around 50 users. One of these services is twiki, a wiki
clone. The system administrators are subscribed to the Debian security
alert emailing service, and to most security alert services for
installed software that is not part of Debian, and security updates
are applied promptly.  Twiki, however, did not provide an emailing
security alert service.  The twiki developers provide security alerts
on wiki pages; in that particular incident, they also did try to
contact as many twiki administrators they knew (about 500), but these
were initially only a small fraction of the twiki installations. They
then compiled a larger list to email the alert to, but this was two
weeks late.

It was an error of the twiki developers that they did not provide a
security alert mailing list, and of me and other twiki administrators
that we had not talked earlier with the developers about this problem,
especially since wikis, being web applications, are naturally prone to
vulnerabilities. It was only after the recent incident, which has
resulted in several compromises, that the problem was seriously
discussed (http://TWiki.org/cgi-bin/view/Codev/TWikiSecurityAlertProcess)
and a mailing list has been setup. Since twiki is a helpful and
reasonable community, I believe they would have solved the problem
earlier if I had raised the issue (and if they hadn't, I should have
moved to another wiki).

Acheloos is a low security system. It mainly serves a university
research team. It has very few confidential documents, and their
degree of confidentiality is not such that we'd loose our sleep in the
event of a leak. Break-ins are thus merely nuisances, but serious
ones; they are unplanned and they are the hardest kind of disruption
to fix.

The attacks
-----------

The twiki "Execute Commands With Search" vulnerability was announced
around 13 November.  Acheloos has been the target of attacks since 14
November. Many of these attacks have been unsuccessful, because of
another server, venetikos, which is a proxy to the twiki of acheloos.
Try http://www.hydro.ntua.gr/twiki/bin/view/Main/WebHome, and it seems
as if the twiki is installed at www.hydro.ntua.gr, that is, venetikos.
However, venetikos actually forwards these requests to acheloos, and
returns its answer to the user. Google lists our wiki as being at
www.hydro.ntua.gr and not as www.itia.ntua.gr (acheloos), and this is
a known problem I intended to correct. Nevertheless, it probably
fooled some attackers who managed to install a backdoor on acheloos
and then tried to contact it at venetikos.

However, at least two attacks were successful; acheloos was
compromised the latest at 19 November. The attacker(s) managed to
install a backdoor that gave them a shell prompt as user www-data, and
then downloaded and compiled programs and attempted unsuccessfully to
become root.

Detection and recovery
----------------------

On 22 November, at about 14:00 local time (EET), I noticed that,
while I was typing (logged on via ssh), acheloos occassionally paused
for a few seconds (maybe 3 or more) and then continued, spitting all
my typed letters at once. Since my connection to acheloos was through
100MBit ethernet, this should not happen. I thus investigated for
possible hanged processes, and restarted apache, which was consuming
much memory (a normal situation since most web sites provided by
acheloos run with mod_perl). While examining the process listing, I
noticed a process named "./cgi" which was run by user www-data, and
immediately suspected that it could be a compromise. I checked the log
files and found that, indeed, some strange, obviously malicious
strings had been passed to twiki at 02:08 (am), the time since when
./cgi had been running. Soon afterwards I checked the twiki web site
and found the security alert. I then saved the output of ps, netstat,
and lsof to files, shut down apache, and killed all processes owned by
www-data.

The reason for acheloos' pauses is not yet known. The system load was
low and the swap usage was normal. Netstat indicated a large number of
connections (all from another, trusted machine) in SYN_RECV status. It
is possible that acheloos was undergoing an unrelated DoS attack, or
that there was some other kind of network problem, or that software
run by the attacker was attacking other systems.

A few days before the intrusion was detected, an assistant
administrator had told me about a hanged process owned by www-data,
which we thought was a hanged apache child but showed with no name in
the ps listing. We had killed it without paying any more attention. I
now believe this was malicious software run by the attacker (I found
more items like it after I detected the intrusion), and we were
obviously wrong not to have further investigated the situation.

Since the machine was otherwise well-patched and there was no evidence
that the attacker(s) had managed to elevate privileges, I proceeded my
investigation with the assumption that the system was otherwise clean
and that I could trust my tools. I expect other administrators to
disagree with this assumption, but not to have made it would have
meant downtime for important services besides apache, and much greater
amount of effort by me.  In addition, I believe that if you know your
system, you can _feel_ when something is wrong, and administrator
intuition must be the hardest thing for rootkits to overpower, no
matter how ingenious they are. However, I'm not the most experienced
administrator on the planet, especially in security issues (and this
is the first compromise I've personally dealt with), and it would be
nice to see comments by others.

Back to my investigation. One particularly useful command was

    find / ! -type l \( -user www-data -o -group www-data -o -perm -2 \) \( -mtime -20 -o -ctime -20 \) -exec ls -lhdc {} \;

which finds all files with user www-data or group www-data or which
are world-writable and have been changed in the last 20 days (in fact
what I intended to find was files writeable by www-data that have
changed in the last 20 days, but this would have been more complex and
the above sufficed). I checked the results one by one and found all
files dropped by the attacker(s), namely backdoors and intrusion
programs. I copied all those to another place, deleted them, patched
twiki, and started apache again. I then intended to take the machine
down for a short while, boot it from a CD, and verify checksums from
the network; however, I didn't find a way to easily download the
needed packages from the network while running from a boot CD (but
maybe a combination of apt-get and debsums --root can do it), and I
was content to run debsums normally, after re-downloading all packages
and regenerating all checksums.

All this process sounds simple, and, indeed, it is simple _after_
you've done it. However, there was a total of 21 hours of apache
downtime. I worked for five hours after detecting the compromise, then
went home and slept, which was tremendously helpful, and the following
day it didn't take more than an hour to finish the work. But maybe if
I were more experienced, I might have fixed it in an hour or so.

Conclusions
-----------

An emergency plan would have helped. CERT's Steps for Recovering from
a UNIX or NT System Compromise is nice, but it deals with root
compromises.  This was a much easier one, for which I'd recommend the
following plan (low-security systems; high-security systems are
obviously outside the scope of this report):

   1. Make sure you've actually been compromised, but don't do
      anything beyond that.

   2. Save the output of the following commands somewhere safe:
      ps aux, ps alx, netstat, netstat -pln, lsof.

   3. If there is evidence of root compromise, skip the rest of this
      and instead read CERT's Steps for Recovering from a UNIX or NT
      System Compromise. Otherwise, assume there is no root
      compromise and go on.

   4. Sit down and relax. Switch your screen off. Make some coffee
      and have it with biscuits. Don't do anything for 20 or 30
      minutes.

   5. Think about the steps you should take in order to clean the
      system and restore all affected services.

   6. Find a newsgroup or mailing list where you might ask for
      technical help in case you need it (better done before the
      compromise).

   7. Discuss with management or affected people.

   8. Start recovery.

A hard copy of this plan should be handy. When I discover a
compromise, I no longer trust my machine or its web pages or its
files. It's only psychological, of course. A more detailed technical
recovery plan might also help, but it is difficult to devise one and
it wouldn't be a substitute for administrator experience and comfort
in the usage of the system tools.

Legal notice
------------

Copyright (C) 2004 Antonios Christofides and National Technical University of Athens

Permission is granted to redistribute freely.