[Opensim-dev] Rate Limit DOS Protection

Tue Oct 8 17:10:15 UTC 2013

 I'm sure you could configure nginx to send the X-Forwarded-For header
to the OpenSim based service and then enable the
DOSAllowXForwardedForHeader in the robust.ini to receive the power of
the 'per service' rate limiting while also having the protection at
the proxy level.    That said, I'm happy you found a configuration
that works for you.

Best Regards

Teravus

On Tue, Oct 8, 2013 at 11:43 AM, Michael Emory Cerquoni
<nebadon2025 at gmail.com> wrote:
> I had a look at the Robust.ini.example earlier and I found the instructions
> to be clear and straight forward, I am glad you did infact allow this to be
> completely disabled, at OSgrid we use nginx as a reverse proxy and also to
> do DoS protection and load balancing, without the options to disable this
> completely it would have been a major problem moving forward.
>
>
> On Tue, Oct 8, 2013 at 12:27 PM, Teravus Ovares <teravus at gmail.com> wrote:
>>
>> Just to be clear, and comment/answer some questions.
>>
>> I did add the DOS protection variables to the Robust.ini.example in
>> the [LoginService] section.  The Rate Limit code does 'per connection'
>> velocity counts.   The default rate setting is 5 requests in 10
>> seconds.    If you have a transparent proxy, such as squid, you need
>> to set DOSAllowXForwardedForHeader to true so that the code trusts the
>> X-Forwarded-For header that your proxy adds to the headers.    If you
>> want to turn it off, set DOSMaxRequestsInTimeFrame = 0.       If you
>> want to allow individual clients to do 20 connections in 5 seconds,
>> set DOSMaxRequestsInTimeFrame = 20 and   DOSRequestTimeFrameMS = 5000.
>>   If you want to change how long individual connections are blocked
>> when they go over the rate limit, change DOSForgiveClientAfterMS.
>>
>> Hopefully this helps get you started with the config options.
>>
>> One more thing, this DOS protector is configured and implemented 'per
>> service',    So, if it's implemented in the login service, it's not
>> going to get triggered by the Group Service.   If there's DOS
>> protection on the Group Service and the login service, and a
>> connection gets blocked on the login service, they'll still be able to
>> access the Group Service because they're individually rate limited...
>>  This is for flexibility.    Choosing what the best rate limit is
>> cannot really be done server wide, it has to be based on the needs of
>> the individual service and that's why it was implemented this way.
>> At this point, the only major service that has rate limiting on by
>> default is the login service.
>>
>> I'll be happy to answer more questions and discuss default settings.
>>
>>
>>  Best Regards
>>
>> Teravus
>>
>>
>> On Tue, Oct 8, 2013 at 8:04 AM, James Stallings II
>> <james.stallings at gmail.com> wrote:
>> > As I have often been told by you, Melanie, if I am upgrading and not
>> > auditing, adjusting, and testing my configs in accordance with changes,
>> > I'm
>> > doing it wrong.
>> >
>> > I think that statement probably applies more to those running larger
>> > concerns than anyone else.
>> >
>> > Honestly, Teravus typically does a better job of coding than to produce
>> > work
>> > that does not take such matters of scale into consideration.
>> >
>> > I'm comfortable with whatever he chooses to do, and if it turns out it
>> > isn't
>> > a case of 'one size fits all', I'll make adjustments accordingly.
>> >
>> >
>> > On Tue, Oct 8, 2013 at 7:52 AM, Melanie <melanie at t-data.com> wrote:
>> >>
>> >> I'm worried that people with larger installations will see service
>> >> failures because legit traffic is seen as abusive. This could cause
>> >> issues
>> >> for the lerger grids out there. I don't believe that whatever tenuous
>> >> protection this may offer for small grids and standalones outwieghs the
>> >> potential service impairment it may cause for unsuspecting larger
>> >> grids. Not
>> >> every grid operator reads this list,
>> >>
>> >> So I'd again suggest that we stick to the way we've always done it and
>> >> make the default for new features be "off".
>> >>
>> >> Melanie
>> >>
>> >> On 8 Oct 2013, at 09:31, Teravus Ovares <teravus at gmail.com> wrote:
>> >>
>> >> I understand what you're saying.      It's hard to argue to leave
>> >> people unprotected from attacks, though.    I'm certainly open to
>> >> making the defaults less protective, and, I'm concerned enough about
>> >> it that I'd prefer to leave some protection in place there.
>> >>
>> >> What are your thoughts on that?
>> >>
>> >> Best Regards
>> >>
>> >> Teravus
>> >>
>> >> On Tue, Oct 8, 2013 at 12:41 AM, Melanie <melanie at t-data.com> wrote:
>> >> > Hi,
>> >> >
>> >> > in keeping with our SOP, the defaults provided should be emulating
>> >> > the previous behavior, e.g. NO rate limiting.
>> >> >
>> >> > I would much appreciate if that procedure would be adhered to,
>> >> > unless we vote to abandon it. Users could suffer because they don't
>> >> > expect the default config to change on them.
>> >> >
>> >> > Cheers,
>> >> >
>> >> > Melanie
>> >> >
>> >> > On 08/10/2013 05:42, Teravus Ovares wrote:
>> >> >> Hi there,
>> >> >>
>> >> >> I just wanted to inform -dev that I added some rate limiting DOS
>> >> >> protection classes to use to protect your opensim based services
>> >> >> from
>> >> >> rapid calling.      At the moment, this will be most noticeable in
>> >> >> the
>> >> >> Login Service.    I have, both as an example, and good practice,
>> >> >> applied the Rate limit protection to the login service.    There are
>> >> >> new Configuration options in StandaloneCommon.ini and Robust.ini
>> >> >> that
>> >> >> control how the connections are rate limited and if trusts the
>> >> >> X-Forwarded-For header.    Just for the sake of getting something up
>> >> >> there, I set the defaults to something sane, however they may not
>> >> >> work
>> >> >> for everyone, so it may be wise to take a look at the new
>> >> >> configuration options in the [LoginService] section of your
>> >> >> bin/Robust.ini.example and
>> >> >> /bin/config-include/StandaloneCommon.ini.example AND/OR have
>> >> >> discussions on what would be more sane default options.   There's a
>> >> >> chance that this could affect anyone, so don't neglect to take a
>> >> >> look
>> >> >> at it.
>> >> >>
>> >> >> You may also notice messages on your console and in your logs like:
>> >> >> 21:56:29 - [LOGINDOSPROTECTION]: client: 192.168.1.213 is blocked
>> >> >> for
>> >> >> 120000 milliseconds, X-ForwardedForAllowed status is False,
>> >> >> endpoint:192.168.1.213
>> >> >>
>> >> >> This is an example of the DOS Protection blocking a connection
>> >> >> because
>> >> >> the client went beyond the rate limit.
>> >> >>
>> >> >> The rate limit is defined by X requests in Y period of time and is
>> >> >> implemented in a rolling Y fashion.   It also has a 'forget' period
>> >> >> of
>> >> >> time that will unblock the blocked user.
>> >> >>
>> >> >> At this point, there's one implemented for XMLRPC handlers, one for
>> >> >> GenericHTTPHandlers and a base class for StreamHandlers based on
>> >> >> BaseStreamHandler.
>> >> >>
>> >> >> If you are interested in the code changes, you can check the diff:
>> >> >>
>> >> >>
>> >> >> http://opensimulator.org/viewgit/?a=commitdiff&p=opensim&h=f76cc6036ebf446553ee5201321879538dafe3b2
>> >> >>
>> >> >> There's still more to do, and, here's a start to providing some
>> >> >> modicum of protection on the services.
>> >> >>
>> >> >> If you have any questions, feel free to reply and ask..  or send me
>> >> >> an
>> >> >> e-mail personally.
>> >> >>
>> >> >> Thanks and Best Regards
>> >> >>
>> >> >> Teravus
>> >> >> _______________________________________________
>> >> >> Opensim-dev mailing list
>> >> >> Opensim-dev at lists.berlios.de
>> >> >> https://lists.berlios.de/mailman/listinfo/opensim-dev
>> >> > _______________________________________________
>> >> > Opensim-dev mailing list
>> >> > Opensim-dev at lists.berlios.de
>> >> > https://lists.berlios.de/mailman/listinfo/opensim-dev
>> >> _______________________________________________
>> >> Opensim-dev mailing list
>> >> Opensim-dev at lists.berlios.de
>> >> https://lists.berlios.de/mailman/listinfo/opensim-dev
>> >>
>> >>
>> >> _______________________________________________
>> >> Opensim-dev mailing list
>> >> Opensim-dev at lists.berlios.de
>> >> https://lists.berlios.de/mailman/listinfo/opensim-dev
>> >
>> >
>> >
>> >
>> > --
>> > ===================================
>> > http://osgrid.org/
>> > http://simhost.com
>> > http://twitter.com/jstallings2
>> >
>> >
>> > _______________________________________________
>> > Opensim-dev mailing list
>> > Opensim-dev at lists.berlios.de
>> > https://lists.berlios.de/mailman/listinfo/opensim-dev
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-dev at lists.berlios.de
>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
>
>
> --
> Michael Emory Cerquoni
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev