[Opensim-dev] Rate Limit DOS Protection

Teravus Ovares teravus at gmail.com
Tue Oct 8 20:23:03 UTC 2013 

LoginService,  Very much agreed.   Though, I figured I'd give people 5
attempts in 10 seconds before sending them on their way in case they
improperly type their password a few times quickly.

Inventory Service,   Also very much agreed.   There is no DOS
protection on that service at the moment.

That's why I figured that this was pretty sane and pretty closely
matches the functionality that people expect while still providing
reasonable protection.    There are two things that I can think of
that may get snagged by the DOS protection..     usage of pcampbot for
load testing.   Clearly that's a rapid login of many clients and is
similar to a DOS attack in many ways.   And people with
reverse/transparent proxies.    That was the reason that I set up the
'AllowXForwardedFor' option.     It's off by default now because if
it's on by default, the DOS protection is easily gamed with the right
header.     It can be turned on and the issue goes away for those
users.

Best Regards

Teravus

On Tue, Oct 8, 2013 at 3:05 PM, Melanie <melanie at t-data.com> wrote:
> The login server should only ever see one connection per client, any more is most likely a DOS.
>
> Services that are called by regions are way more critical, as they get all requests from a few entities only. The inventory service, for instance, can't be rate limited because a single login creates a slew of requests. Also, it should not normally be accessible to the world.
>
> Melanie
>
> On 8 Oct 2013, at 18:27, Teravus Ovares <teravus at gmail.com> wrote:
>
> Just to be clear, and comment/answer some questions.
>
> I did add the DOS protection variables to the Robust.ini.example in
> the [LoginService] section.  The Rate Limit code does 'per connection'
> velocity counts.   The default rate setting is 5 requests in 10
> seconds.    If you have a transparent proxy, such as squid, you need
> to set DOSAllowXForwardedForHeader to true so that the code trusts the
> X-Forwarded-For header that your proxy adds to the headers.    If you
> want to turn it off, set DOSMaxRequestsInTimeFrame = 0.       If you
> want to allow individual clients to do 20 connections in 5 seconds,
> set DOSMaxRequestsInTimeFrame = 20 and   DOSRequestTimeFrameMS = 5000.
>  If you want to change how long individual connections are blocked
> when they go over the rate limit, change DOSForgiveClientAfterMS.
>
> Hopefully this helps get you started with the config options.
>
> One more thing, this DOS protector is configured and implemented 'per
> service',    So, if it's implemented in the login service, it's not
> going to get triggered by the Group Service.   If there's DOS
> protection on the Group Service and the login service, and a
> connection gets blocked on the login service, they'll still be able to
> access the Group Service because they're individually rate limited...
> This is for flexibility.    Choosing what the best rate limit is
> cannot really be done server wide, it has to be based on the needs of
> the individual service and that's why it was implemented this way.
> At this point, the only major service that has rate limiting on by
> default is the login service.
>
> I'll be happy to answer more questions and discuss default settings.
>
>
> Best Regards
>
> Teravus
>
>
> On Tue, Oct 8, 2013 at 8:04 AM, James Stallings II
> <james.stallings at gmail.com> wrote:
>> As I have often been told by you, Melanie, if I am upgrading and not
>> auditing, adjusting, and testing my configs in accordance with changes, I'm
>> doing it wrong.
>>
>> I think that statement probably applies more to those running larger
>> concerns than anyone else.
>>
>> Honestly, Teravus typically does a better job of coding than to produce work
>> that does not take such matters of scale into consideration.
>>
>> I'm comfortable with whatever he chooses to do, and if it turns out it isn't
>> a case of 'one size fits all', I'll make adjustments accordingly.
>>
>>
>> On Tue, Oct 8, 2013 at 7:52 AM, Melanie <melanie at t-data.com> wrote:
>>>
>>> I'm worried that people with larger installations will see service
>>> failures because legit traffic is seen as abusive. This could cause issues
>>> for the lerger grids out there. I don't believe that whatever tenuous
>>> protection this may offer for small grids and standalones outwieghs the
>>> potential service impairment it may cause for unsuspecting larger grids. Not
>>> every grid operator reads this list,
>>>
>>> So I'd again suggest that we stick to the way we've always done it and
>>> make the default for new features be "off".
>>>
>>> Melanie
>>>
>>> On 8 Oct 2013, at 09:31, Teravus Ovares <teravus at gmail.com> wrote:
>>>
>>> I understand what you're saying.      It's hard to argue to leave
>>> people unprotected from attacks, though.    I'm certainly open to
>>> making the defaults less protective, and, I'm concerned enough about
>>> it that I'd prefer to leave some protection in place there.
>>>
>>> What are your thoughts on that?
>>>
>>> Best Regards
>>>
>>> Teravus
>>>
>>> On Tue, Oct 8, 2013 at 12:41 AM, Melanie <melanie at t-data.com> wrote:
>>>> Hi,
>>>>
>>>> in keeping with our SOP, the defaults provided should be emulating
>>>> the previous behavior, e.g. NO rate limiting.
>>>>
>>>> I would much appreciate if that procedure would be adhered to,
>>>> unless we vote to abandon it. Users could suffer because they don't
>>>> expect the default config to change on them.
>>>>
>>>> Cheers,
>>>>
>>>> Melanie
>>>>
>>>> On 08/10/2013 05:42, Teravus Ovares wrote:
>>>>> Hi there,
>>>>>
>>>>> I just wanted to inform -dev that I added some rate limiting DOS
>>>>> protection classes to use to protect your opensim based services from
>>>>> rapid calling.      At the moment, this will be most noticeable in the
>>>>> Login Service.    I have, both as an example, and good practice,
>>>>> applied the Rate limit protection to the login service.    There are
>>>>> new Configuration options in StandaloneCommon.ini and Robust.ini that
>>>>> control how the connections are rate limited and if trusts the
>>>>> X-Forwarded-For header.    Just for the sake of getting something up
>>>>> there, I set the defaults to something sane, however they may not work
>>>>> for everyone, so it may be wise to take a look at the new
>>>>> configuration options in the [LoginService] section of your
>>>>> bin/Robust.ini.example and
>>>>> /bin/config-include/StandaloneCommon.ini.example AND/OR have
>>>>> discussions on what would be more sane default options.   There's a
>>>>> chance that this could affect anyone, so don't neglect to take a look
>>>>> at it.
>>>>>
>>>>> You may also notice messages on your console and in your logs like:
>>>>> 21:56:29 - [LOGINDOSPROTECTION]: client: 192.168.1.213 is blocked for
>>>>> 120000 milliseconds, X-ForwardedForAllowed status is False,
>>>>> endpoint:192.168.1.213
>>>>>
>>>>> This is an example of the DOS Protection blocking a connection because
>>>>> the client went beyond the rate limit.
>>>>>
>>>>> The rate limit is defined by X requests in Y period of time and is
>>>>> implemented in a rolling Y fashion.   It also has a 'forget' period of
>>>>> time that will unblock the blocked user.
>>>>>
>>>>> At this point, there's one implemented for XMLRPC handlers, one for
>>>>> GenericHTTPHandlers and a base class for StreamHandlers based on
>>>>> BaseStreamHandler.
>>>>>
>>>>> If you are interested in the code changes, you can check the diff:
>>>>>
>>>>> http://opensimulator.org/viewgit/?a=commitdiff&p=opensim&h=f76cc6036ebf446553ee5201321879538dafe3b2
>>>>>
>>>>> There's still more to do, and, here's a start to providing some
>>>>> modicum of protection on the services.
>>>>>
>>>>> If you have any questions, feel free to reply and ask..  or send me an
>>>>> e-mail personally.
>>>>>
>>>>> Thanks and Best Regards
>>>>>
>>>>> Teravus
>>>>> _______________________________________________
>>>>> Opensim-dev mailing list
>>>>> Opensim-dev at lists.berlios.de
>>>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>> _______________________________________________
>>>> Opensim-dev mailing list
>>>> Opensim-dev at lists.berlios.de
>>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>> _______________________________________________
>>> Opensim-dev mailing list
>>> Opensim-dev at lists.berlios.de
>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>
>>>
>>> _______________________________________________
>>> Opensim-dev mailing list
>>> Opensim-dev at lists.berlios.de
>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>
>>
>>
>>
>> --
>> ===================================
>> http://osgrid.org/
>> http://simhost.com
>> http://twitter.com/jstallings2
>>
>>
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-dev at lists.berlios.de
>> https://lists.berlios.de/mailman/listinfo/opensim-dev
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev

More information about the Opensim-dev mailing list