Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008540opensim[REGION] OpenSim Corepublic2019-06-05 04:152019-06-09 13:15
Reporterunregi 
Assigned To 
PrioritynormalSeverityfeatureReproducibilityN/A
Statuspatch includedResolutionopen 
PlatformOSOS Version
Product Version0.9.0.1 
Target VersionFixed in Version 
Summary0008540: Add configuration option for differing public Simulator port
DescriptionAttached patch adds a http_public_port option additional to http_listener_port, this allows the simulator and caps to be run behind an http reverse proxy like nginx.
http_public_port defaults to http_listener_port
Additional InformationI will maybe also work on an equivalent patch for ssl, allowing a reverse proxy to do the ssl communication for opensim, faster and more efficient. There is already some unused code for https_external in the HttpServerBase.
Just the
> ; the main unsecure port will still open for some services. this may change in future.
put me off a bit.
What services are meant by that?
TagsNo tags attached.
Git Revision or version number
Run ModeStandalone (1 Region)
Physics EngineBulletSim
EnvironmentMono / Linux64
Mono Version5.x
Viewer
Attached Filespng file icon firestorm-ssl.png [^] (401,454 bytes) 2019-06-09 09:56

- Relationships

-  Notes
(0035273)
tampa (reporter)
2019-06-05 04:45

"this allows the simulator and caps to be run behind an http reverse proxy like nginx" That is already possible without a patch actually
(0035275)
unregi (reporter)
2019-06-05 04:58

How?
The SeedCap returns the URIs with the Port the sim is Listening on.
(0035276)
melanie (administrator)
2019-06-05 05:13

Could you explain the setup that needs this? You could, after all, simply keep the ports the same on the outside and inside.

I'm interested in the setup that requires this.
(0035277)
unregi (reporter)
2019-06-05 05:45

If its a sim with just one or two users and a decent connection, putting it behind nginx on the same machine doesn't give you any noticeable benefits.

It starts to show a better performance with many concurrent connections or/and the connections are slow. Rather than letting a slow texture request blocking up mono, it can just let nginx do the waiting for it. In my specific case, having nginx in front of opensim did solve an issue where i couldn't log in with a large render distance and fullscreen (my sim is on a slow connection).

But the real benefit would be to let nginx handle the SSL connections. That's my end goal, to have the whole sim behind SSL, without having to deal with bad mono HTTPS server performance.
(0035278)
melanie (administrator)
2019-06-05 05:59

Nginx can be configured to accept the connection on a port and forward it to another machine on the same port.

I do seem to see a desire to run nginx on the same machine as opensim and that would indeed require a separate port setting. However, I have another suggestion that would make this work without any core networking change, namely running nginx and the opensim instances in Docker containers. That way ports can remain the same inside and out.

In this instance I'm a bit worried about confusing simple users and making the networking configuration even harder instead of simplifying it.

Having a sim behind SSL is a difficult proposition when using your approach. The reason is that, as far as I am aware, there isn't much support for doing sim to sim and services to sim comms over SSL. A lot of code would have to be fixed or even written to allow that. Also, the current services architecture doesn't differentiate between the port viewers talk to and the port services like the login service talk to.
IMHO, the port for connections from viewers, the port for connections from central services and the port for connections from other simulators should be separate, so that different policies can be applied to them. It's a can of worms. Care for a can opener?
(0035279)
tampa (reporter)
2019-06-05 06:13

While I am not a fan of docker or containers that is what I was referring to, though I have realized it mainly through virtualization rather than containerization. Result is the same. I will say though, I find both ways defeat the push for optimization as they add overhead.

I have to agree with melanie on the can of worms, yet I still feel there eventually should be a push to encrypted connections all around, but as she rightly said this constitutes a rewrite of some of the core networking potentially causing compatibility issues and it would increase performance requirements with all the de-encrypting going on. Famously those changes can bring entire networks to a halt.
(0035280)
unregi (reporter)
2019-06-05 06:48

I am the opposite of tampa, i like docker and appreciate it very much in some cases, but i would not use it just for being able to increase opensim performance a bit with nginx, which can as well be solved with that little patch here, that isn't changing all that much.

SSL requires more work and lots of testing, that's why i thought that i would first make this mantis here with just the basic http port and nothing more.
In case of SSL, i think that the can of worms has to be touched sooner or later anyway (hinting at that unused https_external that's lurking).

Does anybody know what services are meant with:
> ; the main unsecure port will still open for some services. this may change in future.
The region-to-region communication on tps maybe?

And yes, the network configuration is already a bit confusing.
We have BaseHostname, BaseURL, PublicPort, PrivatePort in [Const], then http_listener_port, http_listener_ssl and a bunch of ssl options (which may or may not replace http_listener_port when enabled) and InternalAddress, InternalPort and ExternalHostName in Regions.ini
(so an ExternalPost in Regions.ini maybe would be better instead of a http_public_port in OpenSim.ini for my proposal?)
(0035281)
unregi (reporter)
2019-06-05 06:52

However, if you decide to accept the patch or not, if you decide to do some other changes or not ... please keep that mantis open, i might use it as my personal blog when i try to get as much as possible behind SSL
(0035282)
melanie (administrator)
2019-06-05 07:14

I really believe that one should leverage modern technologies to solve these issues. The "payload port differs from real port" approach seems hacky to me. Then, of course, having IPs and ports in payloads is bad juju anyway and we have grown to hate it massively.

The processing overhead of a docker container is minuscule because docker leverages extensive kernel support. CGroups and IPTables mean that containers run on Linux require no proxying or user space data movement at all.

Running a custom net within Docker instead of using docker0 will also give name resolution inside the system, making the nginx config a no brainer. All without needing to change the port mapping in a quirky, nontransparent way.

I'm running that and other, more complex configs. Works like a charm!
(0035283)
unregi (reporter)
2019-06-05 07:45

Sorry, but having the web application run on a different port than the webserver that is handling the request (doesn't matter if nginx or apache or IIS) is standard.
That is what node applications do and what .NET Core applications do if they use nginx. 10 years ago you connected your application with fastcgi or similar to the webserver, today it's more common to use http in the first place and let the webserver act as reverse proxy.
Even if you would be crazy enough to let the client directly connect to the web application, you would still run it on a different higher port, and make a redirect to it with iptables, because ports <1024 are not available for users on linux.

Containers are solving a different problem and are used for a different task. They are used for dynamic scaling, to provide a sandbox, to be able to sell a million different configurations without bothering with VMs (or just to sell cheaper servers to the customer). Nobody uses containers only for the purpose of being able to use the same ports.

Back to the initial problem....
even if i would use Docker, in case of SSL the problem is still exactly the same and it needs changes in the same parts... because the URIs of the SeedCap also include the protocol http:// and would have to change
(0035284)
melanie (administrator)
2019-06-05 08:01

You're correct. However, this isn't a web server / web app scenario and also there is still webpack/browserify and webserver based openapi handling (which has advantages, like sharing the session id from the regular site).

The difference between these modern web tech protocols and opensim is that those protocols don't usually have IP and port as part of the payload.

I don't believe this should go in as a standalone fix, because it addresses one person's needs. It has never before come up, not even in OSGrid, where nginx is used as well.

I could, however, see it as a stepping stone on a roadmap to a generalization of the handling of ports and IPs in opensim, a root and branch review of the use of the http server and the state of https code.

For that, such a roadmap should exist.
(0035293)
unregi (reporter)
2019-06-05 11:56

Well, sending the URIs to the viewers is what we are stuck with, and there are lots of configuration options on different places already that change those, the port is the only thing that hasn't some config.
(0035294)
unregi (reporter)
2019-06-05 11:58

Attached a patch that removes the unused code of https_external, which, even if it would get used, wouldn't work in its current stage, because it doesn't change the protocol in the SeedCap URIs.
(0035295)
melanie (administrator)
2019-06-05 12:05

Would it not, at this stage, if implementation of https is desired be better to change the CAPS URLs rather than removing the whole feature?
(0035296)
unregi (reporter)
2019-06-05 12:33

lol, yes
but it tells you where it is, in case you want to clean up the code in core and remove it ;)

I think it's current implementation, if fixed, is not very useable, because the configuration would be very confusing... it would make opensim launch a HTTP server on the http_listener_sslport for most services if http_listner_ssl is also true and the user is supposed to put that behind a proxy thats terminating ssl for it... but also the http_listener_port is open for the few services that don't work with ssl.

In my opinion the configuration for https_external have to be seperated from the integrated ssl options.

I would instead make an https_external that just sets the CAPS URIs to https and does nothing more, while the regular server is launched at http_listener_port like usual and a https_external_port that tells it the public HTTPS port of nginx.
This would mean that the CAPS are available via http and https at the same time.
(0035301)
tampa (reporter)
2019-06-05 13:34

This may be extreme, but I would vote for eventually dropping http entirely and moving on to https. Yes breaking compatibility and all, but for it to coexist I find that may not be the best approach for usability and it solves nothing if all users elect to not setup https. I know, I know, forcing is not nice, but if/when/how I feel it may be best to just rewrite the current caps straight to https, to hell with http :)
(0035302)
melanie (administrator)
2019-06-05 13:48

That won't work until someone can figure out how to make the viewer trust letsencrypt. Currently it doesn't.
(0035305)
tampa (reporter)
2019-06-05 13:57

You know more about that than I do, maybe you could see if Dayturn is willing to implement or take a look at supporting that perhaps. If not, if you could point me to a resource that defines what needs implementing I may be able to get a feature request through.
(0035311)
melanie (administrator)
2019-06-05 14:14

The viewer appears to come with it's own certificate selection. It should instead use the system's cert store. Windows and Mac come with a good set of certificates and Linux usually comes with Mozilla's root package. There is no point in shipping roots with the opensource viewer.
(0035313)
unregi (reporter)
2019-06-06 04:32

added a tiny change for the Asset Caps in the patch and reuploaded it
(0035319)
unregi (reporter)
2019-06-07 00:56

Added patch that implements https_external and https_public_port as said before.

Was testing a standalone sim using it with nginx as reverse proxy handling the ssl session with a self-signed certificate (debug option NoVerifySSLCert set in firestorm).
The whole communication was running over https, not a single package used http.

On Hypergrid, tp out worked and i could rezz things from the inventory there (which already implies that the target region used http to fetch the assets, so i didn't even bother with monitoring the traffic), tp in did throw the typical "muh, self-signed cert" error.

There are some confusions in grid_info when https_external is true, but that should be rather easy to track down. And the standalone region is communicating with it's own asset server that it runs itself via HTTPS, which is really dumb.
(0035320)
unregi (reporter)
2019-06-07 01:00

Added patch that bypasses all cert checks, it's just there to show what to temporarily touch when testing regions with self-signed certificates.
(0035321)
melanie (administrator)
2019-06-07 02:22

So are these patches cumulative? First the public port, then ssl external? or has that been combined?
(0035322)
unregi (reporter)
2019-06-07 04:14

That's in order, not combined.
And the next once will be too, because touching the GridInfo and whatever else will probably influence Robust and grid stuff, which i can't really test, so that should be separated imho.

The two small patches now ... or three, because i guess the new https_external patch won't apply if the patch for removing the old https_external isn't applied first ... are rather save, it's more like a clean-up of the https code that's already there.
(0035323)
melanie (administrator)
2019-06-07 04:55

I'll see if I can find the time today. Your tackling this work is much appreciated.
I won't apply the HTTPS verification disable patch, but I have a better one that allows to specify a root certificate to validate against in addition to the system certificate store.
While clients still would have to turn verification off, that would allow full SSL with self-signed certs.
(0035327)
unregi (reporter)
2019-06-08 02:52
edited on: 2019-06-08 02:53

ui, maybe you committed yours already, because i just saw that i got something wrong and that NoVerifyCertChain = true in OpenSim.ini is totally enough to allow self-signed certificate, it even says it in the comment ^^

So i can delete that patch...

(0035328)
unregi (reporter)
2019-06-08 02:59

Regarding the grid_info service.... also this one is totally fine in OpenSim already.

It is Firestorm that can't handle https URLs in the Add-Grid box.
Firestorm tries to connect to the https url with http, gets the nginx error back that this is a ssl port, and then, because the answer isn't making sense for firestorm, assumes some shitty default values.
If i add the grid with the http url and the http port, it works perfectly fine and it gets the https URIs.
(0035329)
melanie (administrator)
2019-06-08 03:01

Yes, that bug is there since the original Hippo grid manager.
(0035352)
EmperorStarfinder (reporter)
2019-06-09 04:38

Actually, I can confirm that Firestorm does handle HTTPS URLs in the Add-Grid box. Has been proven to work with later versions of Firestorm (5.0.11 and 6.0.3). There have been no errors relating to Firestorm not trusting LetsEncrypt certificates either that pop-up or in the viewer's logs.

My team runs a number of internal test grids for our testing purposes. The architectures we use include OpenSim and a number of its forks (or derivatives). I use Firestorm on each fork and have configured the architectures to use SSL certificates including LetsEncrypt and have not run into problems using Firestorm on those grids.

There are, however, issues with installing LetsEncrypt on Windows-based servers but this is mostly because LetsEncrypt doesn't currently have a setup system for Windows which can install and do auto renewals of the certificates as far as I know.
(0035353)
unregi (reporter)
2019-06-09 10:04

1. Firestorm makes http request on https url that's in the Add-Grid box
2. Gets 400 error back, because it's an ssl port
3. Firestorm then connects, this time with ssl, to old legacy cgi-bin/login.cgi
4. fails because its not legacy

Attached logs
You can of course either answer the legacy request or the non-ssl http request, but that's not the goal here.
(0035354)
unregi (reporter)
2019-06-09 11:25

Wrote a bug report to firestorm
https://jira.phoenixviewer.com/browse/FIRE-24068 [^]

I also wanted to ask them to include letsencrypt certificate, but i checked the ca-bundle.crt that firestorm includes, and https://letsencrypt.org/certs/isrgrootx1.pem.txt [^] seems to be inside, so letsencrypt should work, i just can't test it right now.
(0035355)
unregi (reporter)
2019-06-09 11:39

Removed the three patches, because i forgot to adjust the uploader CAPS url and i would have to change the patch again. I don't bother about that, will post a combined patch once i have tps and hypergriding with ssl right.
(the only issues that i see that are left are that the Gatekeeper URI wants to be http, and that HGFriendsModule and EstateConnector have hardcoded http and i bet LSLHttp could also be confused)
(0035357)
UbitUmarov (administrator)
2019-06-09 13:15

all this things will need a major revision to proper implement ssl ( ie tls )

- Issue History
Date Modified Username Field Change
2019-06-05 04:15 unregi New Issue
2019-06-05 04:15 unregi File Added: 0001-Add-http_public_port-option-to-be-able-to-have-a-dif.patch
2019-06-05 04:16 unregi Status new => patch included
2019-06-05 04:45 tampa Note Added: 0035273
2019-06-05 04:58 unregi Note Added: 0035275
2019-06-05 05:13 melanie Note Added: 0035276
2019-06-05 05:45 unregi Note Added: 0035277
2019-06-05 05:59 melanie Note Added: 0035278
2019-06-05 06:13 tampa Note Added: 0035279
2019-06-05 06:48 unregi Note Added: 0035280
2019-06-05 06:52 unregi Note Added: 0035281
2019-06-05 07:14 melanie Note Added: 0035282
2019-06-05 07:45 unregi Note Added: 0035283
2019-06-05 08:01 melanie Note Added: 0035284
2019-06-05 11:12 unregi File Added: 0001-remove-ssl_external.patch
2019-06-05 11:56 unregi Note Added: 0035293
2019-06-05 11:58 unregi Note Added: 0035294
2019-06-05 12:05 melanie Note Added: 0035295
2019-06-05 12:33 unregi Note Added: 0035296
2019-06-05 13:34 tampa Note Added: 0035301
2019-06-05 13:48 melanie Note Added: 0035302
2019-06-05 13:57 tampa Note Added: 0035305
2019-06-05 14:14 melanie Note Added: 0035311
2019-06-06 04:29 unregi File Added: 0001-Add-http_public_port-option-for-a-public-port-thats-.patch
2019-06-06 04:30 unregi File Deleted: 0001-Add-http_public_port-option-to-be-able-to-have-a-dif.patch
2019-06-06 04:32 unregi Note Added: 0035313
2019-06-07 00:33 unregi File Added: 0001-add-better-https_external.patch
2019-06-07 00:56 unregi Note Added: 0035319
2019-06-07 00:57 unregi File Added: 0001-Temporary-deactivate-cert-check.patch
2019-06-07 01:00 unregi Note Added: 0035320
2019-06-07 01:08 unregi File Deleted: 0001-add-better-https_external.patch
2019-06-07 01:09 unregi File Added: 0001-add-better-https_external.patch
2019-06-07 02:22 melanie Note Added: 0035321
2019-06-07 04:14 unregi Note Added: 0035322
2019-06-07 04:55 melanie Note Added: 0035323
2019-06-08 02:52 unregi Note Added: 0035327
2019-06-08 02:53 unregi File Deleted: 0001-Temporary-deactivate-cert-check.patch
2019-06-08 02:53 unregi Note Edited: 0035327 View Revisions
2019-06-08 02:59 unregi Note Added: 0035328
2019-06-08 03:01 melanie Note Added: 0035329
2019-06-09 04:38 EmperorStarfinder Note Added: 0035352
2019-06-09 09:56 unregi File Added: firestorm-ssl.png
2019-06-09 10:04 unregi Note Added: 0035353
2019-06-09 11:25 unregi Note Added: 0035354
2019-06-09 11:31 unregi File Deleted: 0001-remove-ssl_external.patch
2019-06-09 11:32 unregi File Deleted: 0001-Add-http_public_port-option-for-a-public-port-thats-.patch
2019-06-09 11:32 unregi File Deleted: 0001-add-better-https_external.patch
2019-06-09 11:39 unregi Note Added: 0035355
2019-06-09 13:15 UbitUmarov Note Added: 0035357


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker