Security Guideance

[nanog at lacutt.com (LaDerrick H.)]

On Tue, Feb 23, 2010 at 02:46:54PM -0500, Paul Stewart wrote:
> Hi folks...
> 
> 
> 
> We have a strange series of events going on in the past while.... Brief
> history here, looking for input from the community - especially some of
> the security folks on here.
> 
> 
> 
> We provide web hosting services - one of our hosting boxes was found a
> while back with root kits installed, un patched software and lots of
> other "goodies".    With some staff changes in place (don't think I need
> to elaborate on that) we are trying to clean up several issues including
> this particular server.  A new server was provisioned, patched, and
> deployed.  User data was moved over and now the same issue is coming
> back....
> 
> 
> 
> The problem is that a user on this box appears to be launching high
> traffic DOS attacks from it towards other sites.  These are UDP based
> floods that move around from time to time - most of these attacks only
> last a few minutes.

Counting outbound udp bytes and packets can help spot anomalies.
Something like this would help but may be unwieldy if you have thousands
of users on a single box:

WANIF=eth0
userlist="userA userB user..."
for i in ${userlist}
do
   iptables -N ${i}_UDP
   iptables -I OUTPUT -m owner -o ${WANIF} -p udp --uid-owner ${i} -j ${i}_UDP
done

Then look at counters with:
iptables -nvL OUTPUT | grep _UDP | sort.......

I wouldn't leave this in place full-time for thousands of accounts
though without attempting to measure the impact on network performance.

> 
> 
> 
> I've done tcpdumps within seconds of the attack starting and to date
> been unable to find the source of this attack (we know the server,
> just not sure which customer it is on the server that's been
> compromised).  Several hours of scanning for php, cgi, pl type files
> have been wasted and come up nowhere...
> 
> 
> 
> It's been suggested to dump IDS in front of this box and I know I'll
> get some feedback positive and negative in that aspect.
> 
> 
> 
> What tools/practices do others use to resolve this issue?  It's  a
> Centos 5.4 box running latest Plesk control panel.
> 
> 
> 
> Typically we have found it easy to track down the offending script or
> program - this time hasn't been easy at all...
> 
> 
> 
> Thanks,
> 
> 
> 
> Paul
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> ----------------------------------------------------------------------------
> 
> "The information transmitted is intended only for the person or entity
> to which it is addressed and contains confidential and/or privileged
> material. If you received this in error, please contact the sender
> immediately and then destroy this transmission, including all
> attachments, without copying, distributing or disclosing same. Thank
> you."