<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Once again, I am not suggesting a single serial reliable connection, but a reliable connection in addition to the lossy up stream. I understand what you and MIC are saying, I have a strong background in networking. Many packets would still be best used on a lossy UDP stream. However, every time you have a packet that needs to be acknowledged, or resent, that packet is a candidate for a serial stream. If it were so time sensitive that a TCP stream would hurt the remote users experience, then it should be reevaluated whether that packet should be resent in the first place.</div><div class=""><br class=""></div><div class="">A tcp socket is not a new protocol, it would use all existing serialization/deserialization mechanisms for existing mechanisms. It is simply a socket that performs acknowledgements and resends at the hardware level, instead of forcing the application to do so.</div><div class=""><br class=""></div><div class="">I would define a reliable packet as a packet that would be resent until it is confirmed that the client has received it. I am concerned why I have multiple people replying as though I want to rip out the networking udp/http stack and replace it with a single tcp socket. is it something I said?</div><div class=""><br class=""></div><div class="">Michael Heilmann</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class="">Date: Fri, 14 Nov 2014 13:33:51 -0800<br class="">From: Dahlia Trimble <<a href="mailto:dahliatrimble@gmail.com" class="">dahliatrimble@gmail.com</a>><br class="">To: <a href="mailto:opensim-dev@opensimulator.org" class="">opensim-dev@opensimulator.org</a><br class="">Subject: Re: [Opensim-dev] Modifying the networking stack<br class="">Message-ID:<br class=""> <<a href="mailto:CAAQTD4XMSZL90JKqtbabg8wF6rhst9F=kqPVWTOuQ1UNgQf5mQ@mail.gmail.com" class="">CAAQTD4XMSZL90JKqtbabg8wF6rhst9F=kqPVWTOuQ1UNgQf5mQ@mail.gmail.com</a>><br class="">Content-Type: text/plain; charset="utf-8"<br class=""><br class="">I think the term "reliability" needs to be re-evaluated here. As stated in<br class="">earlier messages, the point is to transfer state in a timely manner and not<br class="">waste time and network resources transferring stale, irrelevant state<br class="">messages. Hence, a "reliable stream" transport which can only send serial<br class="">data and does it over a lossy connection must recover all data and deliver<br class="">it in order to the end point and as such is *unreliable* in terms of<br class="">transferring state in a timely manner. I've seen some MMO games which use<br class="">TCP for transferring state and some of them seem to do it quite well but<br class="">they also use a lot of client-side prediction to mitigate the effects of<br class="">lossy networks. Unfortunately many of these prediction techniques often<br class="">fail in a system with dynamic content such as OpenSimulator provides; i.e.,<br class="">one cannot reliably predict a impending collision with an object that has<br class="">not yet rezzed in the client.<br class=""><br class="">While it's been stated that Object Updates form the majority of these<br class="">state-transferring messages, it is not the case that they are the only<br class="">messages. AgentUpdate messages form the majority of messages sent from the<br class="">viewer to the simulator and they are unreliable as adding any reliability<br class="">to them would introduce delays which would significantly reduce quality of<br class="">experience for users. Imagine pressing a button to move and having to wait<br class="">1-2 seconds to move, or imagine playing a battle game and pressing a fire<br class="">button only to have your weapon not fire until several seconds later. There<br class="">are also other messages such as animation state which also need rapid state<br class="">transmission.<br class=""><br class="">Adding a new protocol also involves encoding the data you want and being<br class="">able to decode it when received. Often such encoding/decoding methods can<br class="">themselves be a source of significant overhead. The existing serialization<br class="">code used in libomv (used in OpenSimulator) is fairly efficient at UDP<br class="">packet encoding and decoding and also provides an efficient LLSD (Linden<br class="">Lab Structured Data) codec which is used for the messages sent via<br class="">EventQueueGet. EventQueueGet is also designed to traverse firewalls as it<br class="">uses HTTP long poll techniques. There has also been much work done in<br class="">OpenSimulator to pool and reuse these message structures, thereby reducing<br class="">the overhead of object creation/destruction and garbage collection,<br class=""><br class="">Assuming one *could* have a TCP connection that has a perfect transport<br class="">layer, there is still a single, serial stream which can cause other issues.<br class="">Once you send several messages you cannot immediately send a time critical<br class="">message and have it received until all other messages have been processed<br class="">at the receiving end. While this may be somewhat mitigated by using<br class="">alternate means of transmitting the initial scene as Mic eluded to in his<br class="">proxy example. There will likely also be other times such messages will be<br class="">needed however.<br class=""><br class="">LLUDP protocol used to send assets via UDP in a reliable nature. This had<br class="">the advantage that many assets could be sent simultaneously and<br class="">continuously re-prioritized as the user would move around in the region.<br class="">Unfortunately this required a fair amount of resources on the simulator end<br class="">and was eventually replaced with a HTTP transport which could be offloaded<br class="">from the simulator to another server such as Apache running on another core<br class="">or machine. This HTTP solution was never refined to fully implement the<br class="">dynamic reprioritization or mass simultaneous transfers that the UDP<br class="">transport allowed but has eventually resulted in a usable technology. There<br class="">was a time a while back when competing viewer projects would increase the<br class="">number of simultaneous HTTP downloads in an effort to be the<br class="">fastest-rezzing viewer however SL had problems with their HTTP servers<br class="">being overloaded and had to implement limits. I believe the HTTP RFC<br class="">specification also has a fairly low number of simultaneous connections as a<br class="">recommendation.<br class=""><br class="">I'd also second the recommendation to use WinGridProxy for examining the<br class="">protocols used by OpenSimulator. It is a very effective and useful tool.<br class=""><br class="">On Fri, Nov 14, 2014 at 9:38 AM, Mic Bowman <<a href="mailto:cmickeyb@gmail.com" class="">cmickeyb@gmail.com</a>> wrote:<br class=""><br class=""><blockquote type="cite" class="">Hi Michael!<br class=""><br class="">Pulling out what should already be http over tcp based (textures &<br class="">inventory & profiles & ...)...<br class=""><br class="">Just to be clear... talking very specifically about object updates only<br class="">(though terrain updates are similar in nature), you DO NOT want TCP if your<br class="">error rates are over 1%. you'll waste a huge amount of bw with<br class="">network-level retransmission of old, out-dated updates. Its specifically in<br class="">situations where you have high update rates that you want to send current<br class="">object state on re-transmission rather than re-transmitting an old packet<br class="">that contains an update that has already been superceded.<br class=""><br class="">That being said... the place where UDP is the wrong protocol is the<br class="">initial scene load. It looks a lot like a bunch of one-time object update<br class="">packets where the state of the object does not change.<br class=""><br class="">The approach I spent some time looking at was to create a local scene<br class="">proxy... run an opensim instance on your local client that you connect to<br class="">once... then that instance mirrors remote scenes (kind of like the dynamic<br class="">scene demonstration at oscc except that the scenes are coming from existing<br class="">scenes rather than an archive). The proxy idea was inspired by the<br class="">architecture that Stefan Anderson used years ago in their opensim service<br class="">(I can't remember what it was called).<br class=""><br class="">The proxy design assumes that making major protocol changes to the viewer<br class="">is really hard & expensive, and that local accesses are fast and highly<br class="">reliable. The proxy would enable custom protocols for connection to the<br class="">remote scene which would be a lot easier to optimize (e.g. bulk transfer of<br class="">an oar-like file for initial scene load). The proxy idea got far enough to<br class="">show that it could work for a "consumption" experience... but it would take<br class="">a lot of work to hook up enough events to make it completely transparent.<br class=""><br class="">--mic<br class=""><br class=""><br class=""><br class="">On Fri, Nov 14, 2014 at 9:11 AM, Michael Heilmann <<a href="mailto:mheilman@ist.ucf.edu" class="">mheilman@ist.ucf.edu</a>><br class="">wrote:<br class=""><br class=""><blockquote type="cite" class="">I agree completely with the fact that many packets are time-sensitive<br class="">and expire quickly, which is why I was looking for a TCP/UDP pair, not<br class="">dropping UDP in favour of TCP. I am concerned, however, that UDP is the<br class="">only mechanism for asynchronously sending information to the client that<br class="">may be important, and not expire quickly.<br class=""><br class="">We regularly deal with users who have less then reliable networks, such<br class="">as cellular wifi access points at remote locations, and cannot assume that<br class="">our packet loss will be below 1%.<br class=""><br class="">--<br class="">Michael Heilmann<br class="">Research Associate<br class="">Institute for Simulation and Training<br class="">University of Central Florida<br class=""><br class=""><br class=""><br class="">Date: Fri, 14 Nov 2014 08:30:07 -0800<br class="">From: Mic Bowman <<a href="mailto:cmickeyb@gmail.com" class="">cmickeyb@gmail.com</a>> <<a href="mailto:cmickeyb@gmail.com" class="">cmickeyb@gmail.com</a>><br class="">To: <a href="mailto:opensim-dev@opensimulator.org" class="">opensim-dev@opensimulator.org</a><br class="">Subject: Re: [Opensim-dev] Modifying the networking stack<br class="">Message-ID:<br class=""> <<a href="mailto:CAJaF1_HXXSTFj2KhxDXH4-f_8aSYnWon+QON-NWUn1DommLh4A@mail.gmail.com" class="">CAJaF1_HXXSTFj2KhxDXH4-f_8aSYnWon+QON-NWUn1DommLh4A@mail.gmail.com</a>> <<a href="mailto:CAJaF1_HXXSTFj2KhxDXH4-f_8aSYnWon+QON-NWUn1DommLh4A@mail.gmail.com" class="">CAJaF1_HXXSTFj2KhxDXH4-f_8aSYnWon+QON-NWUn1DommLh4A@mail.gmail.com</a>><br class="">Content-Type: text/plain; charset="utf-8"<br class=""><br class="">having done a lot of the token bucket & retransmission work in opensim... i<br class="">started with a strong belief that TCP & reliable streams were the right way<br class="">to do object updates (for all bulk content like asset transfers, tcp, or<br class="">http layered on tcp, is definitely the right way). however... what we found<br class="">is that object updates should not be retransmitted at the packet level<br class="">which is what you get from tcp (that is... tcp handles retransmission of<br class="">dropped packets not opensim). the reason is pretty simple... dynamic<br class="">objects change position repeatedly (even if only in short bursts).<br class="">retransmission at the packet level would almost always retransmit **OLD**<br class=""><br class="">state. for any kind of connection (this was often a problem for<br class="">international connections) where packet drops were even as high as 1%, we<br class="">would see multiple updates for the same object queued to send. only the<br class="">last one has any meaning.<br class=""><br class="">so... while the situations where application level retransmission are rare<br class="">(**very** rare)... object updates happens to be one of them. moving<br class="">reliability into a lower level (either layering some protocol on tcp or os<br class="">implemented reliable udp) will result in lower efficiency for faulty<br class="">connections (and if your connection isnt faulty then udp works just fine).<br class=""><br class="">--mic<br class=""><br class=""><br class="">On Fri, Nov 14, 2014 at 7:17 AM, Amit Goel <<a href="mailto:Amit.Goel@ucf.edu" class="">Amit.Goel@ucf.edu</a>> <<a href="mailto:Amit.Goel@ucf.edu" class="">Amit.Goel@ucf.edu</a>> wrote:<br class=""><br class=""><br class="">Agree with Michael here that having TCP is better then using UDP with your<br class="">own homegrown TCP implementation on top of it in application software.<br class=""><br class="">How about Reliable UDP :<br class=""><a href="http://www.streamingmedia.com/Articles/Editorial/Featured-Articles/Reliable-UDP-(RUDP)-The-Next-Big-Streaming-Protocol-85316.aspxhttp://www.javvin.com/protocolRUDP.html" class="">http://www.streamingmedia.com/Articles/Editorial/Featured-Articles/Reliable-UDP-(RUDP)-The-Next-Big-Streaming-Protocol-85316.aspxhttp://www.javvin.com/protocolRUDP.html</a><br class=""><br class="">I have not studied it in detail that how it fits in between lossy UDP and<br class="">time-consuming TCP.<br class=""><br class="">Regards<br class=""><br class="">-- amit<br class="">________________________________________<br class="">From: <a href="mailto:opensim-dev-bounces@opensimulator.org" class="">opensim-dev-bounces@opensimulator.org</a> [<a href="mailto:opensim-dev-bounces@opensimulator.org" class="">opensim-dev-bounces@opensimulator.org</a>] on behalf of Michael Heilmann [<a href="mailto:mheilman@ist.ucf.edu" class="">mheilman@ist.ucf.edu</a>]<br class="">Sent: Friday, November 14, 2014 9:23 AM<br class="">To: <a href="mailto:opensim-dev@opensimulator.org" class="">opensim-dev@opensimulator.org</a><br class="">Subject: Re: [Opensim-dev] Modifying the networking stack<br class=""><br class="">Thanks for the responses. I'll go into a little more detail:<br class=""><br class="">We have been running several profilers against OpenSimulator on the<br class="">MOSES grid, and on my development machine. The tests were to examine<br class="">the loading on the server under several different loads, specifically<br class="">mesh and physics loads. What we found appears to be that no matter what<br class="">kind of load we placed on the region, even to the point of becoming<br class="">unresponsive due to physics and mesh, that scripting and physics load<br class="">were nowhere near the amount of time spent in<br class="">OpenSim.Region.ClientStack.LindenUDP once we had more than one or two<br class="">avatars logged in. We know from previous investigations at our firewall<br class="">that network traffic for OpenSim is not that heavy, especially with low<br class="">numbers of users.<br class=""><br class="">I ran several Wireshark captures against a Firestorm viewer logging into<br class="">the MOSES public grid ABWIS region, where we hold our office hours. I<br class="">saw that with our current configuration, all traffic between the server<br class="">and my client, with the exception of http CAPS and fsapi calls, were UDP<br class="">traffic. This is not immediately concerning, as we have simian serve<br class="">our mesh and textures directly. The messages are mostly binary<br class="">information, so I could not examine closely, but I did see a lot of<br class="">messages containing identical ASCII strings, such as the name of my avatar.<br class=""><br class="">My primary concern is the amount of time spent handling networking, not<br class="">necessarily the networking its-self. But there is at least a portion of<br class="">messages on the UDP pipeline that are either reliable, or perhaps should<br class="">be; and re-implementing a reliable transport over udp introduces load at<br class="">the application layer, instead of letting a low-level reliable transport<br class="">such as tcp handle it. I went to university with a guy who implemented<br class="">a java networking library completely over UDP, believing that it was<br class="">faster than a normal TCP socket; but he was neglecting that the<br class="">networking hardware handles the ACK and retransmission transparently,<br class="">and without needing for the messages to be handled manually by the<br class="">application.<br class=""><br class="">This may just be my opinion, but since I was going to be ecamining the<br class="">network stack anyways, and typically in a client-server scenario the<br class="">ability to maintain a persistent reliable connection where the server<br class="">can push important events to the client, that it would be a good idea.<br class="">The points about network throttling and QoS are taken, but wouldn't they<br class="">also typically affect the UDP stream? Working on MOSES I have plenty of<br class="">problems dealing with external users who operate on restricted networks,<br class="">and they cannot see traffic aside from 80 and 443 without dealing with<br class="">their own IT personnel. The fact that it is HTTP over TCP instead of<br class="">raw TCP makes no difference once it is on a non-standard HTTP port.<br class=""><br class="">I agree that it would be more prudent to look at improving the websocket<br class="">code and the http server, rather than replace it with a raw TCP socket,<br class="">especially given that there are multiple plugins, such as jsonsimstats,<br class="">that use the http functionality directly.<br class=""><br class="">I hope that explains my position a little better. I would love to hear<br class="">if there are other plans/ideas in the community to address time-sinks<br class="">like this one, networking simply appears to us as a good starting point<br class="">to increase performance and scalability of the system.<br class=""><br class="">--<br class="">Michael Heilmann<br class="">Research Associate<br class="">Institute for Simulation and Training<br class="">University of Central Florida<br class=""><br class=""><br class=""><br class=""><br class="">Date: Thu, 13 Nov 2014 13:50:32 -0800<br class="">From: Diva Canto<<a href="mailto:diva@metaverseink.com" class="">diva@metaverseink.com</a>> <<a href="mailto:diva@metaverseink.com" class="">diva@metaverseink.com</a>>To:opensim-<a href="mailto:dev@opensimulator.org" class="">dev@opensimulator.org</a><br class="">Subject: Re: [Opensim-dev] Modifying the networking stack<br class="">Message-ID:<<a href="mailto:546527A8.3040909@metaverseink.com" class="">546527A8.3040909@metaverseink.com</a>> <<a href="mailto:546527A8.3040909@metaverseink.com" class="">546527A8.3040909@metaverseink.com</a>><br class="">Content-Type: text/plain; charset=ISO-8859-1; format=flowed<br class=""><br class="">What problem are you trying to solve? It's hard to comment without<br class="">knowing what you want to achieve, but here are some random observations<br class="">that you may want to take into account.<br class=""><br class="">As far as I remember, the reliable packets are a very small percentage<br class="">of the UDP traffic at this point, so I'm not sure it's worth creating a<br class="">dedicated TCP channel for them. Moving the HTTP traffic to the same TCP<br class="">connection seems like a bad idea, as the HTTP traffic tends to be<br class="">dominated by big data (textures, assets, etc) which would then get in<br class="">the way of the small packets like AgentOnline, etc. I suspect it would<br class="">make the client less responsive than what it is now.<br class=""><br class="">There is already support for WebSockets in OpenSim. It may not be<br class="">complete, so I would encourage you to build on that. I am aware of WebGL<br class="">clients that use WebSockets with OpenSim, and they have the same problem<br class="">as described above: the big data gets in the way of the small packets,<br class="">making the clients less responsive at points. But since WebGL is<br class="">inevitable, your effort is probably best invested in this than in a TCP<br class="">channel.<br class=""><br class="">Best,<br class="">Diva<br class=""><br class="">On 11/13/2014 1:18 PM, Michael Heilmann wrote:<br class=""><br class="">Greetings everyone<br class=""><br class="">I and another MOSES developer are going to be looking at the<br class="">client/server network stack, as well as the processing queue's used<br class="">for incoming and outgoing packets. I am going to see if I can<br class="">implement a client stack on opensim and firestorm that uses the<br class="">traditional TCP/UDP pairing for this type of client<->server<br class="">relationship. I have two thoughts, but I am interested in hearing if<br class="">you have ideas or insight into this particular space.<br class=""><br class="">Idea 1:<br class=""> Add a dedicated tcp port next to the UDP port, and move reliable<br class="">transport transmissions to the tcp port. I am uncomfortable<br class="">increasing the required ports for each region, but the http server is<br class="">in the way. I can look to move all communications from http to a tcp<br class="">socket-server type of deployment, at the expense of simple POST/GET<br class="">operations<br class=""><br class="">idea 2:<br class=""> Look into increasing the performance of the http server of the<br class="">regions, as well as testing/implementing a full websockets<br class="">implementation, and using the websockets upgrade for consistent client<br class="">connections. This could eventually lead to javascript-based clients,<br class="">and does not remove http functionality.<br class=""><br class="">Either idea would see any traffic requiring reliable transport shifted<br class="">off of the current UDP stack, and onto the tcp reliable transport.<br class="">Either idea also will require modifications to a client to match. I,<br class="">and another developer here, would be developing the client code, the<br class="">region code, and testing against a MOSES deployment. As we are MOSES<br class="">developers, we would be working against simian instead of Robust; so<br class="">there would be a gap for regular Robust-based grids.<br class=""><br class="">If you could lend me your opinions about these ideas, the management<br class="">queues, associated problems in opensim, etc, I would really appreciate<br class="">it. We would be working completely in the open on github, and obeying<br class="">all licensing. We would welcome any and all cooperation, and we will<br class="">cooperate ourselves wherever we are welcome, but we are not interested<br class="">in avoiding positive changes to maintain SecondLife compatibility.<br class=""><br class="">Thanks.<br class=""><br class=""><br class="">_______________________________________________<br class="">Opensim-dev mailing <a href="mailto:listOpensim-dev@opensimulator.orghttp" class="">listOpensim-dev@opensimulator.orghttp</a>://<a href="http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev" class="">opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev</a><br class="">_______________________________________________<br class="">Opensim-dev mailing <a href="mailto:listOpensim-dev@opensimulator.orghttp" class="">listOpensim-dev@opensimulator.orghttp</a>://<a href="http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev" class="">opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev</a><br class=""><br class=""><br class="">_______________________________________________<br class="">Opensim-dev mailing list<br class=""><a href="mailto:Opensim-dev@opensimulator.org" class="">Opensim-dev@opensimulator.org</a><br class=""><a href="http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev" class="">http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev</a><br class=""><br class=""><br class=""></blockquote><br class="">_______________________________________________<br class="">Opensim-dev mailing list<br class=""><a href="mailto:Opensim-dev@opensimulator.org" class="">Opensim-dev@opensimulator.org</a><br class=""><a href="http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev" class="">http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev</a><br class=""></blockquote></blockquote></div></body></html>