Home
Reading
Searching
Subscribe
Sponsors
Statistics
Posting
Contact
Spam
Lists
Links
About
Hosting
Filtering
Features Download
Marketing
Archives
FAQ
Blog
 
Gmane
From: Christian Folini <christian.folini <at> time-machine.ch>
Subject: Re: DDOS on the appl level, timeouts and blacklisting
Newsgroups: gmane.comp.apache.mod-security.user
Date: Saturday 16th September 2006 11:08:26 UTC (over 10 years ago)
Hello,

A couple of weeks ago i asked for ideas on a defense against
a type of DoS attacks that could be described as
request delaying on the application level (I do not know the
proper technical term so far): Basically, you just open
a connection to a webserver and then you do everything
to keep the connection open while keeping the webserver in the read
phase. You can spend 300 seconds in the read request
header phase by default on a default apache and there won't be 
a single line in one of the logfiles.

When addressing the problem first, i thought mod_security
may be able to help me. I proposed kind of a watchdog timer
to handle the situation, but this would not be generic enough.

A private exchange with Ryan and Ivan evolved. This directed me
away from mod_security. I have done my homework now and i
return to this list. My message is somewhat offtopic, but i
think the general level of knowhow is very high here, so i would
appreciate your imput.

Here we go:

Apache knows a timeout value that is described at
http://httpd.apache.org/docs/2.0/mod/core.html#timeout

> 1. The total amount of time it takes to receive a GET request.
> 2. The amount of time between receipt of TCP packets on a POST or PUT
request.
> 3. The amount of time between ACKs on transmissions of TCP packets in
responses.

(I tested this, and it seems that the [1] is not the total time, but
also the time between TCP packets too.)

This value is 300 seconds by default and this is very high. It is safe
to lower it to below 60 seconds, maybe even 5 but it does not mitigate
the problem, as there are countless ways to escape the timeout.
Among these is the possibility to slow down the ssl handshake, the
read request header phase, the read request body phase, the file upload
phase and especially sly: do a proper first request and slow down the
second when keepalive is active.

As mentioned previously, apache does not even notice. The apache logfile
is written at the very end of a transaction and even the debug error
loglevel does not reveal anything. There are a few indications in the
ssl-log, but only when slowing down that handshake. The rest
goes all unnoticed. Still the logfiles are the key, as a single apache
thread or process has a very limited scope. See mod_evasive that
protects from DoS attacks, but only as long as the attacker uses
keepalive.
-> A single apache thread or process lacks the "whole picture" and
it can not do a proper assessment of the probability of an attack.

Therefore i propose to defend in 3 seperate steps:
 - reconnaissance (apache log)
 - analysis (external process scanning logfiles in realtime)
 - defense (ip blacklisting or anything else)

It can be argued, that 2 and 3 should be within a single
mechanism, but the important thing is to seperate reconnaissance
and analysis. 

What do we need to notice the request delaying attack when
it is happening? We need information about the state of
a request. This is what mod_status is about, but mod_status
does not tell us the ip address of a client host during the
read phase and furthermore, accessing http://localhost/status
may be a hopeless task during a DoS attack.
So i opt for writing more log messages than apache does
today.

To be more precise, i want to write the following state messages:
 - open connection (pre-connection)
 - finished read request header (post-read-header)
 - finished read request body (access-checker)
 - finished request (log-transaction)
 - close connection (hook missing)

I have listed the respective apache 2.0 hooks in brackets. I have looked
at existing modules as mod_evasive, mod_log_forensic, mod_security
to find the hook suiteable to the task.

The problem is, that there is nothing like a close connection hook.
However, in http://hypermail.linklord.com/new-httpd/2004/Feb/3597.html
i have found a patch from Joe Orton, that brings a finish_connection hook.

At cut down version of the patch looks as follows:

--- httpd-2.0.59/server/connection.c    2006-07-12 09:40:55.000000000 +0200
+++ httpd-2.0.59.patched/server/connection.c    2006-08-31
16:49:37.880730953 +0200
@@ -36,12 +36,15 @@
	     APR_HOOK_LINK(create_connection)
	    APR_HOOK_LINK(process_connection)
	    APR_HOOK_LINK(pre_connection)
+           APR_HOOK_LINK(finish_connection)
 )
 AP_IMPLEMENT_HOOK_RUN_FIRST(conn_rec *,create_connection,
			     (apr_pool_t *p, server_rec *server, apr_socket_t *csd, long
conn_id, void *sbh, apr_bucket_alloc_t *alloc),
			     (p, server, csd, conn_id, sbh, alloc), NULL)
 AP_IMPLEMENT_HOOK_RUN_FIRST(int,process_connection,(conn_rec
*c),(c),DECLINED)
 AP_IMPLEMENT_HOOK_RUN_ALL(int,pre_connection,(conn_rec *c, void *csd),(c,
csd),OK,DECLINED)
+AP_IMPLEMENT_HOOK_RUN_ALL(int,finish_connection,(conn_rec
*c),(c),OK,DECLINED)
+
 /*
  * More machine-dependent networking gooo... on some systems,
  * you've got to be *really* sure that all the packets are acknowledged
@@ -174,6 +177,7 @@
     
     if (!c->aborted) {
	 ap_run_process_connection(c);
+       ap_run_finish_connection(c);
     }
 }
 
--- httpd-2.0.59/include/http_connection.h      2006-07-12
09:40:55.000000000 +0200
+++ httpd-2.0.59.patched/include/http_connection.h      2006-08-31
16:50:25.069198657 +0200
@@ -104,6 +104,19 @@
  */
 AP_DECLARE_HOOK(int,process_connection,(conn_rec *c))
 
+/**
+ * This hook allows connection-level filters to perform any necessary
+ * processing before a connection is closed.
+ * @param c The connection which is about to be closed
+ * @return OK or DECLINED
+ * @deffunc int ap_run_finish_connection(conn_rec *c)
+ */
+AP_DECLARE_HOOK(int,finish_connection,(conn_rec *c))
+
+#ifdef __cplusplus
+}
+#endif
+
 /* End Of Connection (EOC) bucket */
 
 AP_DECLARE_DATA extern const apr_bucket_type_t ap_bucket_type_eoc;






With this, we get all the necessary hooks to write a new logging
module that writes all necessary data to the error_log (as a starting
point).

I have called this module mod_log_reqphase as a working title.
Below is the main part of the module:


static int log_reqphase_pre_conn(conn_rec *c, void *csd)
{
    ap_log_error(APLOG_MARK, APLOG_INFO, 0, c->base_server,
"mod_log_reqphase: init connection from client %s, port %u", c->remote_ip,
c->remote_addr->port);
    
    return OK;
}

static int log_reqphase_post_read_request(request_rec *r)
{
    ap_log_error(APLOG_MARK, APLOG_INFO, 0, r->connection->base_server,
"mod_log_reqphase: finished reading headers from client %s, port %u",
r->connection->remote_ip, r->connection->remote_addr->port);
    
    return OK;
}

static int log_reqphase_log_transaction(request_rec *r)
{
    ap_log_error(APLOG_MARK, APLOG_INFO, 0, r->connection->base_server,
"mod_log_reqphase: finished request from client %s, port %u",
r->connection->remote_ip, r->connection->remote_addr->port);
    
    return OK;
}

static int log_reqphase_finish_connection(conn_rec *c)
{
    ap_log_error(APLOG_MARK, APLOG_INFO, 0, c->base_server,
"mod_log_reqphase: closed connection from client %s, port %u",
c->remote_ip, c->remote_addr->port);
    
    return OK;
}

static void log_reqphase_register_hooks(apr_pool_t *p)
{

  ap_hook_pre_connection(log_reqphase_pre_conn, NULL, NULL,
APR_HOOK_MIDDLE);
 
ap_hook_post_read_request(log_reqphase_post_read_request,NULL,NULL,APR_HOOK_MIDDLE);
 
ap_hook_log_transaction(log_reqphase_log_transaction,NULL,NULL,APR_HOOK_MIDDLE);
 
ap_hook_finish_connection(log_reqphase_finish_connection,NULL,NULL,APR_HOOK_MIDDLE);

}

The access checker is not yet present in the code piece above. That's
because i just did a proof of concept so far.


Now that we get the necessary infos, how can we analyse them to detect
the attack?

- attack #1: open a connection, remain idle
  detection: the analyzer notices a read-request-header phase not being
   finished n seconds after the opening of the connection
- attack #2: delaying the ssl handshake
  detection: the analyzer notices a read-request-header phase not being
   finished n seconds after the opening of the connection
- attack #3: delaying the request header read phase by sending in one
(bogus-)
   header after other, each one in a single tcp packet
  detection: the analyzer notices a read-request-header phase not being
   finished n seconds after the opening of the connection
- attack #4: delaying the request body read phase
  detection: the analyzer notices the phase is not done within n
   seconds. Optionally an additional mod_security check
   after the read header phase to make sure a post is feasible to the
   URL in question at all. Actually in a standard application, an
non-authorized 
   client should only be able to make a post to the login-form. 
   However, it is a bit tricky to detect the client being unauthorised 
   in mod_security after the read request header phase already.
   All i can see is a check against a list of client ip addresses with
   open sessions, provided by the authentication layer of the
   application running.
- attack #5: Delaying a file upload
  detection: File uploads can be big. I would try to check with
  mod_security wether the client is authorised to upload at all.
  Otherwise this may be quite tricky for the external analyzer.
  How to tell a slow modem over a weak connection from a DoS attacker...
- attack #6: Using keepalive to delay a subsequent request
  detection: After a finished request, the connection has to be closed
  after k seconds or a new read header phase has to be finished after
  k + n seconds.

I do not want to dive into the actual defense once a client is
identified as an attacker. However, i would have this log analysis
running during normal operation and try to tune it to minimize
the false positives. Then once a serious attack is starting, the
defense action can be triggered either automatically or by hand.

I do not know, if this makes any sense at all. I wonder what you guys 
think and welcome any feedback.
The point is, that network layer DoS attacks are getting harder and
harder and i see the application level getting more attention.
And i think the application level is not yet prepared in the case
of apache.

best regards,

Christian Folini

-- 
First you make it, then it works, then you invite people to 
make it better. 
-- Eben Moglen, Free Software Foundation

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job
easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
 
CD: 3ms