<div dir="ltr">If Singularity is doing one AgentUpdate per second then things have changed quite a bit since I did my measurements with GridProxy a few years back. I'd also question the need to reduce them further if the frequency is that low. My measurements with LL viewers had a AgentUpdate rate that correspended with viewer frame rate and would often exceed 60 frames AND update per second. <br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 18, 2014 at 4:03 PM, Justin Clark-Casey <span dir="ltr"><<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Currently, pCampbot turns off the libomv asset cache and has its own texture/mesh cache. This is currently very similar - there's only one cache for all bots although we mere flag our receipt of the textures/mesh and don't keep the actual data. Of course, this behaviour can change in the future.<br>
<br>
Default agent update in libomv is currently 2 per second by default (Settings.DEFAULT_AGENT_<u></u>UPDATE_INTEVAL). From memory, I thought I observed Singularity post about 1 per second but I could be wrong (Diva may know what it really is now). That was only a couple of observations so may change under different conditions or different viewers.<span class=""><br>
<br>
On 18/11/14 22:35, Dahlia Trimble wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
One issue with libomv bots (and I'm not sure if this applies to pCampbot or not) is that running multiple bots from the<br>
same installation of libomv results in them all sharing the same asset cache so asset fetches by such clients will be<br>
much lower than normal viewers, perhaps even by an order of magnitude or more.<br>
<br>
Another issue is that libomv AgentUpdate runs at a fixed rate of 10/second but many viewers send at a rate which is<br>
roughly the same as frame draw rate. Bots in general don't really need a high AgentUpdate rate and 10/second is probably<br>
a good choice. While I agree with Diva that high rates in general are undesirable due to the extra work they impose on<br>
the region UDP stack, I question how much they can be reduced, or even made "smart" without degrading user experience<br>
while interacting in a region over a slow or lossy network connection. Some user applications such as games or<br>
interactive simulations may require fast response to controls and could suffer if such needs are not considered while<br>
engineering a networking system.<br>
<br></span><div><div class="h5">
On Tue, Nov 18, 2014 at 1:51 PM, Justin Clark-Casey <<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.com</a> <mailto:<a href="mailto:jjustincc@googlemail.com" target="_blank">jjustincc@googlemail.<u></u>com</a>>> wrote:<br>
<br>
These are all fixable issues, either with pCampbot improvements or distributing pCampbot instances amongst more<br>
machines. I expect pCampbot will be built upon to address these points as required. And this year I successfully<br>
used 4 Amazon c2 large instances for bot running so a more realistic network load means spinning up more cloud<br>
instances.<br>
<br>
I agree that unless you can reproduce an issue you are shooting in the dark with any changes. And organizing enough<br>
real people to reproduce issues on a regular basis and without a huge amount of confusing other behaviour is<br>
impossible in practice.<br>
<br>
<br>
On 14/11/14 16:46, Maxwell, Douglas CIV USARMY ARL (US) wrote:<br>
<br>
Classification: UNCLASSIFIED<br>
Caveats: NONE<br>
<br>
Dr. Lopez, thank you for sharing your paper. Can you tell me where it was<br>
peer reviewed and published? I would like to reference it in my<br>
dissertation.<br>
<br>
On the topic of bots, the MOSES team has not been able to compose a NPC<br>
agent or bot that accurately replicate the footprint of a human agent on the<br>
simulator. We believe this is for many reasons:<br>
<br>
1) Bots are usually composed on a server on the same network, not dispersed<br>
across the internet. The bots should be software throttled and noise<br>
introduced into their connections to approximate random access.<br>
<br>
2) Bots aren't using full clients, so they are not filling caches and<br>
making the same scene requests as humans in graphical clients.<br>
<br>
3) Bots are usually homogenous. They need to be randomly dressed, have<br>
random attachments, and have random inventories.<br>
<br>
4) Bots need to move randomly and collide with objects in the scene and<br>
with each other.<br>
<br>
5) Bots need to randomly chat with each other and broadcast locally.<br>
<br>
We think we can create a NPC solution that satisfies these issues. Will<br>
take some thought and development. Has anyone come close to this?<br>
<br>
Goal: Compose bots/NPCs that can approximate the loads of humans within 90%<br>
certainty. Meaning if we load 100 of these artificial agents into the<br>
MOSES, we are certain that it will accurately behave as if at least 90<br>
humans are logged in.<br>
<br>
IMHO, if you can't assign a reliability to a test, then you are just wasting<br>
your time. This is basic V&V tenants.<br>
<br>
v/r -douglas<br>
<br>
Douglas Maxwell, MSME<br>
Science and Technology Manager<br>
Virtual World Strategic Applications<br>
U.S. Army Research Lab<br>
Simulation & Training Technology Center (STTC)<br></div></div>
(c) <a href="tel:%28407%29%20242-0209" value="+14072420209" target="_blank">(407) 242-0209</a> <tel:%28407%29%20242-0209><span class=""><br>
<br>
<br>
<br>
-----Original Message-----<br>
From: opensim-dev-bounces@__<a href="http://opensimulator.org" target="_blank">opensimu<u></u>lator.org</a> <mailto:<a href="mailto:opensim-dev-bounces@opensimulator.org" target="_blank">opensim-dev-bounces@<u></u>opensimulator.org</a>><br>
[mailto:<a href="mailto:opensim-dev-bounces@" target="_blank">opensim-dev-bounces@</a>__<a href="http://opensimulator.org" target="_blank"><u></u>opensimulator.org</a> <mailto:<a href="mailto:opensim-dev-bounces@opensimulator.org" target="_blank">opensim-dev-bounces@<u></u>opensimulator.org</a>>] On Behalf Of<br>
Diva Canto<br>
Sent: Friday, November 14, 2014 11:05 AM<br></span><span class="">
To: <a href="mailto:opensim-dev@opensimulator.org" target="_blank">opensim-dev@opensimulator.org</a> <mailto:<a href="mailto:opensim-dev@opensimulator.org" target="_blank">opensim-dev@<u></u>opensimulator.org</a>><br>
Subject: Re: [Opensim-dev] Modifying the networking stack<br>
<br>
On 11/14/2014 6:23 AM, Michael Heilmann wrote:<br>
<br>
Thanks for the responses. I'll go into a little more detail:<br>
<br>
We have been running several profilers against OpenSimulator on the<br>
MOSES grid, and on my development machine. The tests were to examine<br>
the loading on the server under several different loads, specifically<br>
mesh and physics loads. What we found appears to be that no matter<br>
what kind of load we placed on the region, even to the point of<br>
becoming unresponsive due to physics and mesh, that scripting and<br>
physics load were nowhere near the amount of time spent in<br></span>
OpenSim.Region.ClientStack.__<u></u>LindenUDP once we had more than one or two<span class=""><br>
avatars logged in. We know from previous investigations at our<br>
firewall that network traffic for OpenSim is not that heavy,<br>
especially with low numbers of users.<br>
<br>
<br>
If this is a problem, and you are running a recent-ish version of core<br>
OpenSim, it sounds like some misconfiguration somewhere. Back in the summer<br>
of 2013 we had a problem with the server running OSCC'13; the kernel was<br>
configured to run in some sort of special mode that was making everything<br>
run badly and unpredictably. We fixed the kernel configuration, and suddenly<br>
things started running much more smoothly-- I don't remember the details,<br>
but Nebadon may clarify things.<br>
<br>
OpenSim these days can handle 50 people on a single simulator without much<br>
trouble. If you look at figure 7 of my paper<br></span>
(<a href="http://www.ics.uci.edu/~__lopes/documents/summersim14/__gabrielova_lopes_prepri" target="_blank">http://www.ics.uci.edu/~__<u></u>lopes/documents/summersim14/__<u></u>gabrielova_lopes_prepri</a><div><div class="h5"><br>
<<a href="http://www.ics.uci.edu/~lopes/documents/summersim14/gabrielova_lopes_prepri" target="_blank">http://www.ics.uci.edu/~<u></u>lopes/documents/summersim14/<u></u>gabrielova_lopes_prepri</a>><br>
nt.pdf)<br>
you will see the quantification of "without much trouble." I suggest that<br>
you reproduce my experimental conditions with pCamBot and check whether your<br>
numbers are very different from ours. If they are very different, then<br>
there's definitely something odd in your setup, as we were able to reproduce<br>
these numbers in several machines. Feel free to contact me directly for<br>
details about pCamBot configuration.<br>
<br>
Bots aren't real viewers, but they are much better for measuring things<br>
systematically and detecting problems and bottlenecks than relying on real<br>
users driven by real people. The performance you get with pCamBot will be<br>
correlated with the performance you get with real users.<br>
<br>
<br>
I ran several Wireshark captures against a Firestorm viewer logging<br>
into the MOSES public grid ABWIS region, where we hold our office<br>
hours. I saw that with our current configuration, all traffic between<br>
the server and my client, with the exception of http CAPS and fsapi<br>
calls, were UDP traffic. This is not immediately concerning, as we<br>
have simian serve our mesh and textures directly. The messages are<br>
mostly binary information, so I could not examine closely, but I did<br>
see a lot of messages containing identical ASCII strings, such as the<br>
name of my avatar.<br>
<br>
<br>
Hard to say what you saw, but I bet those are the AgentUpdate messages that<br>
I mentioned before. The viewer sends at least 10/sec. At points, the viewer<br>
sends much more than 10/sec, up to 60/sec. Again, take a look at my paper<br>
for understanding what those are, and how OpenSim deals with them since<br>
OSCC'13.<br>
<br>
As I said before, it would be nice to understand why the viewer is so eager<br>
to blabber its status to the server when nothing is going on.<br>
<br>
<br>
My primary concern is the amount of time spent handling networking,<br>
not necessarily the networking its-self. But there is at least a<br>
portion of messages on the UDP pipeline that are either reliable, or<br>
perhaps should be; and re-implementing a reliable transport over udp<br>
introduces load at the application layer, instead of letting a<br>
low-level reliable transport such as tcp handle it. I went to<br>
university with a guy who implemented a java networking library<br>
completely over UDP, believing that it was faster than a normal TCP<br>
socket; but he was neglecting that the networking hardware handles the<br>
ACK and retransmission transparently, and without needing for the<br>
messages to be handled manually by the application.<br>
<br>
This may just be my opinion, but since I was going to be ecamining the<br>
network stack anyways, and typically in a client-server scenario the<br>
ability to maintain a persistent reliable connection where the server<br>
can push important events to the client, that it would be a good<br>
idea. The points about network throttling and QoS are taken, but<br>
wouldn't they also typically affect the UDP stream? Working on MOSES I<br>
have plenty of problems dealing with external users who operate on<br>
restricted networks, and they cannot see traffic aside from 80 and 443<br>
without dealing with their own IT personnel. The fact that it is HTTP<br>
over TCP instead of raw TCP makes no difference once it is on a<br>
non-standard HTTP port.<br>
<br>
I agree that it would be more prudent to look at improving the<br>
websocket code and the http server, rather than replace it with a raw<br>
TCP socket, especially given that there are multiple plugins, such as<br>
jsonsimstats, that use the http functionality directly.<br>
<br>
I hope that explains my position a little better. I would love to<br>
hear if there are other plans/ideas in the community to address<br>
time-sinks like this one, networking simply appears to us as a good<br>
starting point to increase performance and scalability of the system.<br>
<br>
<br></div></div>
______________________________<u></u>___________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@opensimulator.org" target="_blank">Opensim-dev@opensimulator.org</a> <mailto:<a href="mailto:Opensim-dev@opensimulator.org" target="_blank">Opensim-dev@<u></u>opensimulator.org</a>><br>
<a href="http://opensimulator.org/cgi-__bin/mailman/listinfo/opensim-__dev" target="_blank">http://opensimulator.org/cgi-_<u></u>_bin/mailman/listinfo/opensim-<u></u>__dev</a><span class=""><br>
<<a href="http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev" target="_blank">http://opensimulator.org/cgi-<u></u>bin/mailman/listinfo/opensim-<u></u>dev</a>><br>
<br>
Classification: UNCLASSIFIED<br>
Caveats: NONE<br>
<br>
<br>
<br>
<br></span>
______________________________<u></u>___________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@opensimulator.org" target="_blank">Opensim-dev@opensimulator.org</a> <mailto:<a href="mailto:Opensim-dev@opensimulator.org" target="_blank">Opensim-dev@<u></u>opensimulator.org</a>><br>
<a href="http://opensimulator.org/cgi-__bin/mailman/listinfo/opensim-__dev" target="_blank">http://opensimulator.org/cgi-_<u></u>_bin/mailman/listinfo/opensim-<u></u>__dev</a><span class=""><br>
<<a href="http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev" target="_blank">http://opensimulator.org/cgi-<u></u>bin/mailman/listinfo/opensim-<u></u>dev</a>><br>
<br>
<br>
<br>
--<br>
Justin Clark-Casey (justincc)<br>
OSVW Consulting<br>
<a href="http://justincc.org" target="_blank">http://justincc.org</a><br>
<a href="http://twitter.com/justincc" target="_blank">http://twitter.com/justincc</a><br>
<br></span>
______________________________<u></u>___________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@opensimulator.org" target="_blank">Opensim-dev@opensimulator.org</a> <mailto:<a href="mailto:Opensim-dev@opensimulator.org" target="_blank">Opensim-dev@<u></u>opensimulator.org</a>><br>
<a href="http://opensimulator.org/cgi-__bin/mailman/listinfo/opensim-__dev" target="_blank">http://opensimulator.org/cgi-_<u></u>_bin/mailman/listinfo/opensim-<u></u>__dev</a><br>
<<a href="http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev" target="_blank">http://opensimulator.org/cgi-<u></u>bin/mailman/listinfo/opensim-<u></u>dev</a>><span class=""><br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@opensimulator.org" target="_blank">Opensim-dev@opensimulator.org</a><br>
<a href="http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev" target="_blank">http://opensimulator.org/cgi-<u></u>bin/mailman/listinfo/opensim-<u></u>dev</a><br>
<br>
</span></blockquote><div class="HOEnZb"><div class="h5">
<br>
<br>
-- <br>
Justin Clark-Casey (justincc)<br>
OSVW Consulting<br>
<a href="http://justincc.org" target="_blank">http://justincc.org</a><br>
<a href="http://twitter.com/justincc" target="_blank">http://twitter.com/justincc</a><br>
______________________________<u></u>_________________<br>
Opensim-dev mailing list<br>
<a href="mailto:Opensim-dev@opensimulator.org" target="_blank">Opensim-dev@opensimulator.org</a><br>
<a href="http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev" target="_blank">http://opensimulator.org/cgi-<u></u>bin/mailman/listinfo/opensim-<u></u>dev</a><br>
</div></div></blockquote></div><br></div>