[Opensim-dev] Rate Limit DOS Protection
Teravus Ovares
teravus at gmail.com
Tue Oct 8 20:31:47 UTC 2013
Another note, I can probably expand the DOS protection class to
identify X sessions and establish a maximum number of sessions pretty
easily and with really low overhead. I have not yet.. just X
requests in Y seconds and Z seconds to unblock.
I may do that later this week. (but won't have it active by default)
Best Regards
Teravus
On Tue, Oct 8, 2013 at 3:23 PM, Teravus Ovares <teravus at gmail.com> wrote:
> LoginService, Very much agreed. Though, I figured I'd give people 5
> attempts in 10 seconds before sending them on their way in case they
> improperly type their password a few times quickly.
>
> Inventory Service, Also very much agreed. There is no DOS
> protection on that service at the moment.
>
> That's why I figured that this was pretty sane and pretty closely
> matches the functionality that people expect while still providing
> reasonable protection. There are two things that I can think of
> that may get snagged by the DOS protection.. usage of pcampbot for
> load testing. Clearly that's a rapid login of many clients and is
> similar to a DOS attack in many ways. And people with
> reverse/transparent proxies. That was the reason that I set up the
> 'AllowXForwardedFor' option. It's off by default now because if
> it's on by default, the DOS protection is easily gamed with the right
> header. It can be turned on and the issue goes away for those
> users.
>
> Best Regards
>
> Teravus
>
> On Tue, Oct 8, 2013 at 3:05 PM, Melanie <melanie at t-data.com> wrote:
>> The login server should only ever see one connection per client, any more is most likely a DOS.
>>
>> Services that are called by regions are way more critical, as they get all requests from a few entities only. The inventory service, for instance, can't be rate limited because a single login creates a slew of requests. Also, it should not normally be accessible to the world.
>>
>> Melanie
>>
>> On 8 Oct 2013, at 18:27, Teravus Ovares <teravus at gmail.com> wrote:
>>
>> Just to be clear, and comment/answer some questions.
>>
>> I did add the DOS protection variables to the Robust.ini.example in
>> the [LoginService] section. The Rate Limit code does 'per connection'
>> velocity counts. The default rate setting is 5 requests in 10
>> seconds. If you have a transparent proxy, such as squid, you need
>> to set DOSAllowXForwardedForHeader to true so that the code trusts the
>> X-Forwarded-For header that your proxy adds to the headers. If you
>> want to turn it off, set DOSMaxRequestsInTimeFrame = 0. If you
>> want to allow individual clients to do 20 connections in 5 seconds,
>> set DOSMaxRequestsInTimeFrame = 20 and DOSRequestTimeFrameMS = 5000.
>> If you want to change how long individual connections are blocked
>> when they go over the rate limit, change DOSForgiveClientAfterMS.
>>
>> Hopefully this helps get you started with the config options.
>>
>> One more thing, this DOS protector is configured and implemented 'per
>> service', So, if it's implemented in the login service, it's not
>> going to get triggered by the Group Service. If there's DOS
>> protection on the Group Service and the login service, and a
>> connection gets blocked on the login service, they'll still be able to
>> access the Group Service because they're individually rate limited...
>> This is for flexibility. Choosing what the best rate limit is
>> cannot really be done server wide, it has to be based on the needs of
>> the individual service and that's why it was implemented this way.
>> At this point, the only major service that has rate limiting on by
>> default is the login service.
>>
>> I'll be happy to answer more questions and discuss default settings.
>>
>>
>> Best Regards
>>
>> Teravus
>>
>>
>> On Tue, Oct 8, 2013 at 8:04 AM, James Stallings II
>> <james.stallings at gmail.com> wrote:
>>> As I have often been told by you, Melanie, if I am upgrading and not
>>> auditing, adjusting, and testing my configs in accordance with changes, I'm
>>> doing it wrong.
>>>
>>> I think that statement probably applies more to those running larger
>>> concerns than anyone else.
>>>
>>> Honestly, Teravus typically does a better job of coding than to produce work
>>> that does not take such matters of scale into consideration.
>>>
>>> I'm comfortable with whatever he chooses to do, and if it turns out it isn't
>>> a case of 'one size fits all', I'll make adjustments accordingly.
>>>
>>>
>>> On Tue, Oct 8, 2013 at 7:52 AM, Melanie <melanie at t-data.com> wrote:
>>>>
>>>> I'm worried that people with larger installations will see service
>>>> failures because legit traffic is seen as abusive. This could cause issues
>>>> for the lerger grids out there. I don't believe that whatever tenuous
>>>> protection this may offer for small grids and standalones outwieghs the
>>>> potential service impairment it may cause for unsuspecting larger grids. Not
>>>> every grid operator reads this list,
>>>>
>>>> So I'd again suggest that we stick to the way we've always done it and
>>>> make the default for new features be "off".
>>>>
>>>> Melanie
>>>>
>>>> On 8 Oct 2013, at 09:31, Teravus Ovares <teravus at gmail.com> wrote:
>>>>
>>>> I understand what you're saying. It's hard to argue to leave
>>>> people unprotected from attacks, though. I'm certainly open to
>>>> making the defaults less protective, and, I'm concerned enough about
>>>> it that I'd prefer to leave some protection in place there.
>>>>
>>>> What are your thoughts on that?
>>>>
>>>> Best Regards
>>>>
>>>> Teravus
>>>>
>>>> On Tue, Oct 8, 2013 at 12:41 AM, Melanie <melanie at t-data.com> wrote:
>>>>> Hi,
>>>>>
>>>>> in keeping with our SOP, the defaults provided should be emulating
>>>>> the previous behavior, e.g. NO rate limiting.
>>>>>
>>>>> I would much appreciate if that procedure would be adhered to,
>>>>> unless we vote to abandon it. Users could suffer because they don't
>>>>> expect the default config to change on them.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Melanie
>>>>>
>>>>> On 08/10/2013 05:42, Teravus Ovares wrote:
>>>>>> Hi there,
>>>>>>
>>>>>> I just wanted to inform -dev that I added some rate limiting DOS
>>>>>> protection classes to use to protect your opensim based services from
>>>>>> rapid calling. At the moment, this will be most noticeable in the
>>>>>> Login Service. I have, both as an example, and good practice,
>>>>>> applied the Rate limit protection to the login service. There are
>>>>>> new Configuration options in StandaloneCommon.ini and Robust.ini that
>>>>>> control how the connections are rate limited and if trusts the
>>>>>> X-Forwarded-For header. Just for the sake of getting something up
>>>>>> there, I set the defaults to something sane, however they may not work
>>>>>> for everyone, so it may be wise to take a look at the new
>>>>>> configuration options in the [LoginService] section of your
>>>>>> bin/Robust.ini.example and
>>>>>> /bin/config-include/StandaloneCommon.ini.example AND/OR have
>>>>>> discussions on what would be more sane default options. There's a
>>>>>> chance that this could affect anyone, so don't neglect to take a look
>>>>>> at it.
>>>>>>
>>>>>> You may also notice messages on your console and in your logs like:
>>>>>> 21:56:29 - [LOGINDOSPROTECTION]: client: 192.168.1.213 is blocked for
>>>>>> 120000 milliseconds, X-ForwardedForAllowed status is False,
>>>>>> endpoint:192.168.1.213
>>>>>>
>>>>>> This is an example of the DOS Protection blocking a connection because
>>>>>> the client went beyond the rate limit.
>>>>>>
>>>>>> The rate limit is defined by X requests in Y period of time and is
>>>>>> implemented in a rolling Y fashion. It also has a 'forget' period of
>>>>>> time that will unblock the blocked user.
>>>>>>
>>>>>> At this point, there's one implemented for XMLRPC handlers, one for
>>>>>> GenericHTTPHandlers and a base class for StreamHandlers based on
>>>>>> BaseStreamHandler.
>>>>>>
>>>>>> If you are interested in the code changes, you can check the diff:
>>>>>>
>>>>>> http://opensimulator.org/viewgit/?a=commitdiff&p=opensim&h=f76cc6036ebf446553ee5201321879538dafe3b2
>>>>>>
>>>>>> There's still more to do, and, here's a start to providing some
>>>>>> modicum of protection on the services.
>>>>>>
>>>>>> If you have any questions, feel free to reply and ask.. or send me an
>>>>>> e-mail personally.
>>>>>>
>>>>>> Thanks and Best Regards
>>>>>>
>>>>>> Teravus
>>>>>> _______________________________________________
>>>>>> Opensim-dev mailing list
>>>>>> Opensim-dev at lists.berlios.de
>>>>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>>> _______________________________________________
>>>>> Opensim-dev mailing list
>>>>> Opensim-dev at lists.berlios.de
>>>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>> _______________________________________________
>>>> Opensim-dev mailing list
>>>> Opensim-dev at lists.berlios.de
>>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>>
>>>>
>>>> _______________________________________________
>>>> Opensim-dev mailing list
>>>> Opensim-dev at lists.berlios.de
>>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>
>>>
>>>
>>>
>>> --
>>> ===================================
>>> http://osgrid.org/
>>> http://simhost.com
>>> http://twitter.com/jstallings2
>>>
>>>
>>> _______________________________________________
>>> Opensim-dev mailing list
>>> Opensim-dev at lists.berlios.de
>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-dev at lists.berlios.de
>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>
>>
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-dev at lists.berlios.de
>> https://lists.berlios.de/mailman/listinfo/opensim-dev
More information about the Opensim-dev
mailing list