[Opensim-dev] OpenSim search

Fri Feb 8 09:15:00 UTC 2008

Hi everybody:

Sorry for my bad english, I hope you can understand what I want to say.

Last days I was looking for a way to integrate NHibernate into Opensim. I
have finished some integration tests and now I am working with the current
Database model (8 tables by now) for making a "basic" change of only the
database manager (ADO for NHibernate). If everything goes as I expect the
change doesn't affect anything in the code and we will gain (as I say in
previous posts) things like database independence, security due to
transactions, speed improved by nhibernate caches, etc. etc. etc.

The problem, and that's the point where my investigations joins with yours,
is that we need a better database model. Maybe in the beggining of opensim
the needs were to make the server work, but now the project is mature enough
to have a better database model. We are talking yet of communicating servers
(authorizing users in different servers), to search for things (into the
user inventory, region, server...) and we expect good results, and quick.
And other point is that is a bottleneck right now (just try to copy prims by
thousands) and the problem is not the API, is the model, it isn't flexible
enough for big needs.

I am not a DBA, but I have enough experience developing architectures for
ERP applications (banks, insurance companies, nuclear plants) and I did
enough migrations from bad (or old) database models to new and normalized
(sometimes the problems are resolved just normalizing db) to know the
importance of the persistence in a big application.

And if we want a good search system... why we have to do tricks to convert
data to xml or something (JSON for interserver communications, i.e.) when we
simply need "select" knowing the type of data we are looking for? Just
follow KISS DRY

In resume: I think we need to sit down and redesign the database, the
complete persistence, before integrating more services "that work with the
data present in the sim".

Greetingzzz

2008/2/8, Charles Krinke <cfk at pacbell.net>:
>
> Dear David:
>
> Maybe it might be a good idea if you and Diva were to collaborate a little
> bit. Couldnt hurt.
>
> Charles
>
> ----- Original Message ----
> From: David Wendt JR. <dcrkid at yahoo.com>
> To: opensim-dev at lists.berlios.de
> Sent: Thursday, February 7, 2008 4:00:40 PM
> Subject: Re: [Opensim-dev] OpenSim search
>
>  Yes. I think we came up with this a few days ago on IRC, and I would like
> to start hammering out details. I already have some code to index parcels,
> but right now all it does is output parcel changes to the console since
> there's no API or transport protocol to expose.
>
> A simple exposed XML file polled at set intervals would be fine. But I
> would also like a way for a search service to subscribe to a sim's XML file
> to have it pushed (delta-compressed, of course) to the search service
> whenever something changes or at set intervals such that low-traffic sims
> get updated whenever something changes, but high-traffic sims don't flood
> the service with too many updates.
>
> So I'll be back with an XML schema idea.
>
> ----- Original Message ----
> From: Charles Krinke <cfk at pacbell.net>
> To: diva at metaverseink.com; opensim-dev at lists.berlios.de
> Sent: Thursday, February 7, 2008 1:29:02 PM
> Subject: Re: [Opensim-dev] OpenSim search
>
>  I'll weigh in here and say that I think Diva's thoughts appear sound to
> me and I would like to encourage them and offer a friendly environment to
> test them on OSGrid as they come together.
>
> Charles
>
> ----- Original Message ----
> From: Diva Canto <diva at metaverseink.com>
> To: opensim-dev at lists.berlios.de
> Sent: Thursday, February 7, 2008 9:01:05 AM
> Subject: [Opensim-dev] OpenSim search
>
> Hi all,
>
> I've been spending some time studying the project, to see how exactly I
> can contribute towards enabling search engines in OpenSims/Grids. I have
> a more concrete plan now that I'd like to share with you, and ask
> whether this is something you think it's worth pursuing or not. If not,
> then, well, never mind, it's just a plan! :-) If you think it's worth
> pursuing, after getting your comments I will proceed to coding this
> weekend, and when I'm done I'll submit my code as a patch for your
> assessment and consideration. The patch involved in this is relatively
> independent of everything that already exists, it would simply be
> another server command that doesn't interfere with the existing ones.
> Here's the details.
>
> The very basic for a search engine to exist is the data that lives on
> the sims. (I'll talk more about which data and data access control in
> the next paragraph, but let's ignore that in this paragraph.) As such,
> that data has to be exposed for collection. The right way to do this is
> to generate an XML file representing the information on the sims, namely
> parcel and object information. This file can then be served by the sim
> via, for example, http. So, a possible interface to data collection
> services might be as simple as this: tagus.ics.uci.edu/OpenSim Test.xml.
> My concrete proposal as a first step is to implement a command that
> generates this XML file, possibly called xml-snapshot.
>
> Unlike the existing save/load commands, whose purpose is admin backups
> and data ports, this external exposure doesn't need so many details and
> must be aware of exposure controls.
> - Less details: children prims in objects are irrelevant -- only the
> root prims matter.
> - Exposure control: the viewer's checkbox "Show in search" is a good
> indicator of users intensions towards  external exposure. (In fact, that
> button should really be called "Expose externally"). Only objects and
> parcels that have that marked should be exposed.
>
> Additionally to users intensions, there can be two more layers of
> control, one at the sim admin level, and one at the grid admin level.
> - At the sim level, each sim admin should have the power to override the
> users checkboxes. Specifically, if sim admins decide that nothing in the
> sim should be exposed then the xml-snapshot for those sims is a no-op.
> - At the grid level, each grid operator should have the power to decide
> whether to externally serve common assets, such as textures, or not.
> More importanly, the grid operator should also be able to instruct the
> sims on that grid that they should not serve the XML file to external
> data collectors, only to data collectors that are "authorized" by the
> grid.
>
> These controls enable a variety of grids, from very open to completely
> closed, homogeneous, heterogeneous, and everything in between.
> Once this external data exposure is in place, data collection components
> (search engines and others) can do their thing.
>
> Note that Linden Lab has recently moved from being a closed grid to
> being open, with respect to sim and user data. Unfortunately, they
> decided to use HTML as data representation...duh.
>
> Additionally to the xml-snapshot command, I can volunteer to implement a
> basic Lucene-based search engine that OpenSim grid operators can run on
> their own grids. This is independent of any other search engines, and is
> particularly useful in grids that are closed to the outside. The
> argument here is that Lucene-based text search  is 100x better than
> relational queries, under all aspects I can think of -- run-time
> performance, sim/grid load, component coupling, and relevance of search
> results.
>
> Note that some parts of the search problem can still be served in the
> traditional way. Specifically, user search seems like a really simple
> thing to do with SQL, it's not like there are many Diva Canto's on the
> grid... So the idea of a SQL-based search service for certain kinds of
> data is not incompatible with what I'm proposing.
>
> I'll wait for your comments, and, as I said, if you think this is not
> worth pursuing, I'll go away :-)
> Probably won't be able to reply until later tonight.
>
> Diva
>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
>
> -----Inline Attachment Follows-----
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
> ------------------------------
> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it
> now.<http://us.rd.yahoo.com/evt=51733/*http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ>
>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://opensimulator.org/pipermail/opensim-dev/attachments/20080208/52908836/attachment-0001.html>