[Opensim-dev] OpenSim search

Fri Feb 8 11:48:07 UTC 2008

That the db layer is sorely underdeveloped is not news to anybody (although I haven't been following Seans progress lately) and that we need to reorganize and normalize is clear.

Could anybody that has worked with nhibernate write a letter to opensim-dev and explain how one would go about to, for example, just update a single part field, without cascading changes to child fields (like shape) and, how the object model has to be organized for it to work? (like, for example, can we use accessors, do they need to be public, or can they be protected? can we have customized field types like Vector and Quaternion, or do we have to 'explode' those types ourselves?)

I'm very much into the whole xml discussion, partly because my company, Tribal Media, are actually working with web/OpenSim integration, which means we're already sending world information over http.

In this work, we've seen that there's often ridiculous amounts of duplication in xml and db serialization; fields are duplicated on every level and data that is shared across instances are duplicated.

Getting an RSS subscription feed of object changes has a lot of interesting possibilities... the mind races. :-D

One might dwell a bit on the 'what's actually interesting' bit of the objects, but suffice to say that you really want a (custom) discovery protocol.
Other than that, the web publishing interface should be implemented as a RegionModule (they get the Scene sent to them) so they are optional and keep core impact to a minimum.

Best,
Stefan

Date: Fri, 8 Feb 2008 12:00:50 +0100From: dalienta at gmail.comTo: opensim-dev at lists.berlios.deSubject: Re: [Opensim-dev] OpenSim searchIn the spirit of rfc1925, I think it makes sense to split the database redesign discussion into a separate thread :-)I like Diva's idea of an XML file - this file can be "pushed" elsewhere by means external to opensim. Also very attractive approach from David with the delta-changes.My suggestion on how to combine those two:1) have an API which generates XML of "everything which is exposed and has changed since timestamp X" (and the other constraints according to Diva's mail).2) use this API to generate this static file, as we please - depending on the particular deployment scenario.The reason for suggesting not to have the command is to be able to externally script the invocations of it on as-needed basis.3) for the future - foresee the callback to trigger the re-generation of the files / notifying the search servers accordingly.  Simple algorithm for that: new change starts the timers T1 and T2, with T2 >> T1. Each subsequent change restarts the T1 - not T2. When either of the two timers elapses, push the change through. This will ensure the small or bursty changes get indexed quite quickly (in T1 time), and guarantees that the changes will eventually be indexed (T2 time).Restricting the access to the data being indexed only to some search engines is a matter of simple network access control, imho. (any of the modern web servers does this task fairly well)XMPP is an interesting idea - however, it belongs to an implementation of the search engine, not to the "data supplier" interface on opensim, imho.As usual - the above are merely my musings, I guess it would be useful to hook up on irc with David and Diva and discuss this in more details - and then - the working implementation wins! :-)/d
On Feb 8, 2008 10:15 AM, Impalah <impalah at gmail.com> wrote:
Hi everybody:Sorry for my bad english, I hope you can understand what I want to say.Last days I was looking for a way to integrate NHibernate into Opensim. I have finished some integration tests and now I am working with the current Database model (8 tables by now) for making a "basic" change of only the database manager (ADO for NHibernate). If everything goes as I expect the change doesn't affect anything in the code and we will gain (as I say in previous posts) things like database independence, security due to transactions, speed improved by nhibernate caches, etc. etc. etc.The problem, and that's the point where my investigations joins with yours, is that we need a better database model. Maybe in the beggining of opensim the needs were to make the server work, but now the project is mature enough to have a better database model. We are talking yet of communicating servers (authorizing users in different servers), to search for things (into the user inventory, region, server...) and we expect good results, and quick. And other point is that is a bottleneck right now (just try to copy prims by thousands) and the problem is not the API, is the model, it isn't flexible enough for big needs.I am not a DBA, but I have enough experience developing architectures for ERP applications (banks, insurance companies, nuclear plants) and I did enough migrations from bad (or old) database models to new and normalized (sometimes the problems are resolved just normalizing db) to know the importance of the persistence in a big application.And if we want a good search system... why we have to do tricks to convert data to xml or something (JSON for interserver communications, i.e.) when we simply need "select" knowing the type of data we are looking for? Just follow KISS DRYIn resume: I think we need to sit down and redesign the database, the complete persistence, before integrating more services "that work with the data present in the sim".Greetingzzz
2008/2/8, Charles Krinke <cfk at pacbell.net>:

Dear David:

Maybe it might be a good idea if you and Diva were to collaborate a little bit. Couldnt hurt.

Charles
----- Original Message ----From: David Wendt JR. <dcrkid at yahoo.com>To: opensim-dev at lists.berlios.de
Sent: Thursday, February 7, 2008 4:00:40 PMSubject: Re: [Opensim-dev] OpenSim search

Yes. I think we came up with this a few days ago on IRC, and I would like to start hammering out details. I already have some code to index parcels, but right now all it does is output parcel changes to the console since there's no API or transport protocol to expose.A simple exposed XML file polled at set intervals would be fine. But I would also like a way for a search service to subscribe to a sim's XML file to have it pushed (delta-compressed, of course) to the search service whenever something changes or at set intervals such that low-traffic sims get updated whenever something changes, but high-traffic sims don't flood the service with too many updates.So I'll be back with an XML schema idea.
----- Original Message ----From: Charles Krinke <cfk at pacbell.net>To: diva at metaverseink.com; opensim-dev at lists.berlios.deSent: Thursday, February 7, 2008 1:29:02 PMSubject: Re: [Opensim-dev] OpenSim search

I'll weigh in here and say that I think Diva's thoughts appear sound to me and I would like to encourage them and offer a friendly environment to test them on OSGrid as they come together.

Charles
----- Original Message ----From: Diva Canto <diva at metaverseink.com>To: opensim-dev at lists.berlios.deSent: Thursday, February 7, 2008 9:01:05 AMSubject: [Opensim-dev] OpenSim searchHi all,I've been spending some time studying the project, to see how exactly I can contribute towards enabling search engines in OpenSims/Grids. I have a more concrete plan now that I'd like to share with you, and ask whether this is something you think it's worth pursuing or not. If not, then, well, never mind, it's just a plan! :-) If you think it's worth pursuing, after getting your comments I will proceed to coding this weekend, and when I'm done I'll submit my code as a patch for your assessment and consideration. The patch involved in this is relatively independent of everything that already exists, it would simply be another server command that doesn't interfere with the existing ones. Here's the details.The very basic for a search engine to exist is the data that lives on the sims. (I'll talk more about which data and data access control in the next paragraph, but let's ignore that in this paragraph.) As such, that data has to be exposed for collection. The right way to do this is to generate an XML file representing the information on the sims, namely parcel and object information. This file can then be served by the sim via, for example, http. So, a possible interface to data collection services might be as simple as this: tagus.ics.uci.edu/OpenSim Test.xml. My concrete proposal as a first step is to implement a command that generates this XML file, possibly called xml-snapshot.Unlike the existing save/load commands, whose purpose is admin backups and data ports, this external exposure doesn't need so many details and must be aware of exposure controls.- Less details: children prims in objects are irrelevant -- only the root prims matter.- Exposure control: the viewer's checkbox "Show in search" is a good indicator of users intensions towards  external exposure. (In fact, that button should really be called "Expose externally"). Only objects and parcels that have that marked should be exposed.Additionally to users intensions, there can be two more layers of control, one at the sim admin level, and one at the grid admin level.- At the sim level, each sim admin should have the power to override the users checkboxes. Specifically, if sim admins decide that nothing in the sim should be exposed then the xml-snapshot for those sims is a no-op.- At the grid level, each grid operator should have the power to decide whether to externally serve common assets, such as textures, or not. More importanly, the grid operator should also be able to instruct the sims on that grid that they should not serve the XML file to external data collectors, only to data collectors that are "authorized" by the grid.These controls enable a variety of grids, from very open to completely closed, homogeneous, heterogeneous, and everything in between.Once this external data exposure is in place, data collection components (search engines and others) can do their thing.Note that Linden Lab has recently moved from being a closed grid to being open, with respect to sim and user data. Unfortunately, they decided to use HTML as data representation...duh.Additionally to the xml-snapshot command, I can volunteer to implement a basic Lucene-based search engine that OpenSim grid operators can run on their own grids. This is independent of any other search engines, and is particularly useful in grids that are closed to the outside. The argument here is that Lucene-based text search  is 100x better than relational queries, under all aspects I can think of -- run-time performance, sim/grid load, component coupling, and relevance of search results.Note that some parts of the search problem can still be served in the traditional way. Specifically, user search seems like a really simple thing to do with SQL, it's not like there are many Diva Canto's on the grid... So the idea of a SQL-based search service for certain kinds of data is not incompatible with what I'm proposing.I'll wait for your comments, and, as I said, if you think this is not worth pursuing, I'll go away :-)Probably won't be able to reply until later tonight.Diva_______________________________________________Opensim-dev mailing listOpensim-dev at lists.berlios.dehttps://lists.berlios.de/mailman/listinfo/opensim-dev

-----Inline Attachment Follows-----_______________________________________________Opensim-dev mailing listOpensim-dev at lists.berlios.dehttps://lists.berlios.de/mailman/listinfo/opensim-dev

Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now.
_______________________________________________Opensim-dev mailing listOpensim-dev at lists.berlios.dehttps://lists.berlios.de/mailman/listinfo/opensim-dev_______________________________________________Opensim-dev mailing listOpensim-dev at lists.berlios.dehttps://lists.berlios.de/mailman/listinfo/opensim-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://opensimulator.org/pipermail/opensim-dev/attachments/20080208/48a98c49/attachment-0001.html>