[Opensim-dev] OpenSim search

Fri Feb 8 11:00:50 UTC 2008

In the spirit of rfc1925, I think it makes sense to split the database
redesign discussion into a separate thread :-)

I like Diva's idea of an XML file - this file can be "pushed" elsewhere by
means external to opensim.

Also very attractive approach from David with the delta-changes.

My suggestion on how to combine those two:

1) have an API which generates XML of "everything which is exposed and has
changed since timestamp X" (and the other constraints according to Diva's
mail).
2) use this API to generate this static file, as we please - depending on
the particular deployment scenario.

The reason for suggesting not to have the command is to be able to
externally script the invocations of it on as-needed basis.

3) for the future - foresee the callback to trigger the re-generation of the
files / notifying the search servers accordingly.
  Simple algorithm for that: new change starts the timers T1 and T2, with T2
>> T1. Each subsequent change restarts the T1 - not T2. When either of the
two timers elapses, push the change through. This will ensure the small or
bursty changes get indexed quite quickly (in T1 time), and guarantees that
the changes will eventually be indexed (T2 time).

Restricting the access to the data being indexed only to some search engines
is a matter of simple network access control, imho. (any of the modern web
servers does this task fairly well)

XMPP is an interesting idea - however, it belongs to an implementation of
the search engine, not to the "data supplier" interface on opensim, imho.

As usual - the above are merely my musings, I guess it would be useful to
hook up on irc with David and Diva and discuss this in more details - and
then - the working implementation wins! :-)

/d

On Feb 8, 2008 10:15 AM, Impalah <impalah at gmail.com> wrote:

> Hi everybody:
>
> Sorry for my bad english, I hope you can understand what I want to say.
>
> Last days I was looking for a way to integrate NHibernate into Opensim. I
> have finished some integration tests and now I am working with the current
> Database model (8 tables by now) for making a "basic" change of only the
> database manager (ADO for NHibernate). If everything goes as I expect the
> change doesn't affect anything in the code and we will gain (as I say in
> previous posts) things like database independence, security due to
> transactions, speed improved by nhibernate caches, etc. etc. etc.
>
> The problem, and that's the point where my investigations joins with
> yours, is that we need a better database model. Maybe in the beggining of
> opensim the needs were to make the server work, but now the project is
> mature enough to have a better database model. We are talking yet of
> communicating servers (authorizing users in different servers), to search
> for things (into the user inventory, region, server...) and we expect good
> results, and quick. And other point is that is a bottleneck right now (just
> try to copy prims by thousands) and the problem is not the API, is the
> model, it isn't flexible enough for big needs.
>
> I am not a DBA, but I have enough experience developing architectures for
> ERP applications (banks, insurance companies, nuclear plants) and I did
> enough migrations from bad (or old) database models to new and normalized
> (sometimes the problems are resolved just normalizing db) to know the
> importance of the persistence in a big application.
>
> And if we want a good search system... why we have to do tricks to convert
> data to xml or something (JSON for interserver communications, i.e.) when
> we simply need "select" knowing the type of data we are looking for? Just
> follow KISS DRY
>
> In resume: I think we need to sit down and redesign the database, the
> complete persistence, before integrating more services "that work with the
> data present in the sim".
>
> Greetingzzz
>
>
>
> 2008/2/8, Charles Krinke <cfk at pacbell.net>:
>
> > Dear David:
> >
> > Maybe it might be a good idea if you and Diva were to collaborate a
> > little bit. Couldnt hurt.
> >
> > Charles
> >
> > ----- Original Message ----
> > From: David Wendt JR. <dcrkid at yahoo.com>
> > To: opensim-dev at lists.berlios.de
> > Sent: Thursday, February 7, 2008 4:00:40 PM
> > Subject: Re: [Opensim-dev] OpenSim search
> >
> >  Yes. I think we came up with this a few days ago on IRC, and I would
> > like to start hammering out details. I already have some code to index
> > parcels, but right now all it does is output parcel changes to the console
> > since there's no API or transport protocol to expose.
> >
> > A simple exposed XML file polled at set intervals would be fine. But I
> > would also like a way for a search service to subscribe to a sim's XML file
> > to have it pushed (delta-compressed, of course) to the search service
> > whenever something changes or at set intervals such that low-traffic sims
> > get updated whenever something changes, but high-traffic sims don't flood
> > the service with too many updates.
> >
> > So I'll be back with an XML schema idea.
> >
> > ----- Original Message ----
> > From: Charles Krinke <cfk at pacbell.net>
> > To: diva at metaverseink.com; opensim-dev at lists.berlios.de
> > Sent: Thursday, February 7, 2008 1:29:02 PM
> > Subject: Re: [Opensim-dev] OpenSim search
> >
> >  I'll weigh in here and say that I think Diva's thoughts appear sound to
> > me and I would like to encourage them and offer a friendly environment to
> > test them on OSGrid as they come together.
> >
> > Charles
> >
> > ----- Original Message ----
> > From: Diva Canto <diva at metaverseink.com>
> > To: opensim-dev at lists.berlios.de
> > Sent: Thursday, February 7, 2008 9:01:05 AM
> > Subject: [Opensim-dev] OpenSim search
> >
> > Hi all,
> >
> > I've been spending some time studying the project, to see how exactly I
> > can contribute towards enabling search engines in OpenSims/Grids. I have
> >
> > a more concrete plan now that I'd like to share with you, and ask
> > whether this is something you think it's worth pursuing or not. If not,
> > then, well, never mind, it's just a plan! :-) If you think it's worth
> > pursuing, after getting your comments I will proceed to coding this
> > weekend, and when I'm done I'll submit my code as a patch for your
> > assessment and consideration. The patch involved in this is relatively
> > independent of everything that already exists, it would simply be
> > another server command that doesn't interfere with the existing ones.
> > Here's the details.
> >
> > The very basic for a search engine to exist is the data that lives on
> > the sims. (I'll talk more about which data and data access control in
> > the next paragraph, but let's ignore that in this paragraph.) As such,
> > that data has to be exposed for collection. The right way to do this is
> > to generate an XML file representing the information on the sims, namely
> >
> > parcel and object information. This file can then be served by the sim
> > via, for example, http. So, a possible interface to data collection
> > services might be as simple as this: tagus.ics.uci.edu/OpenSim Test.xml.
> >
> > My concrete proposal as a first step is to implement a command that
> > generates this XML file, possibly called xml-snapshot.
> >
> > Unlike the existing save/load commands, whose purpose is admin backups
> > and data ports, this external exposure doesn't need so many details and
> > must be aware of exposure controls.
> > - Less details: children prims in objects are irrelevant -- only the
> > root prims matter.
> > - Exposure control: the viewer's checkbox "Show in search" is a good
> > indicator of users intensions towards  external exposure. (In fact, that
> >
> > button should really be called "Expose externally"). Only objects and
> > parcels that have that marked should be exposed.
> >
> > Additionally to users intensions, there can be two more layers of
> > control, one at the sim admin level, and one at the grid admin level.
> > - At the sim level, each sim admin should have the power to override the
> >
> > users checkboxes. Specifically, if sim admins decide that nothing in the
> >
> > sim should be exposed then the xml-snapshot for those sims is a no-op.
> > - At the grid level, each grid operator should have the power to decide
> > whether to externally serve common assets, such as textures, or not.
> > More importanly, the grid operator should also be able to instruct the
> > sims on that grid that they should not serve the XML file to external
> > data collectors, only to data collectors that are "authorized" by the
> > grid.
> >
> > These controls enable a variety of grids, from very open to completely
> > closed, homogeneous, heterogeneous, and everything in between.
> > Once this external data exposure is in place, data collection components
> >
> > (search engines and others) can do their thing.
> >
> > Note that Linden Lab has recently moved from being a closed grid to
> > being open, with respect to sim and user data. Unfortunately, they
> > decided to use HTML as data representation...duh.
> >
> > Additionally to the xml-snapshot command, I can volunteer to implement a
> >
> > basic Lucene-based search engine that OpenSim grid operators can run on
> > their own grids. This is independent of any other search engines, and is
> >
> > particularly useful in grids that are closed to the outside. The
> > argument here is that Lucene-based text search  is 100x better than
> > relational queries, under all aspects I can think of -- run-time
> > performance, sim/grid load, component coupling, and relevance of search
> > results.
> >
> > Note that some parts of the search problem can still be served in the
> > traditional way. Specifically, user search seems like a really simple
> > thing to do with SQL, it's not like there are many Diva Canto's on the
> > grid... So the idea of a SQL-based search service for certain kinds of
> > data is not incompatible with what I'm proposing.
> >
> > I'll wait for your comments, and, as I said, if you think this is not
> > worth pursuing, I'll go away :-)
> > Probably won't be able to reply until later tonight.
> >
> > Diva
> >
> >
> > _______________________________________________
> > Opensim-dev mailing list
> > Opensim-dev at lists.berlios.de
> > https://lists.berlios.de/mailman/listinfo/opensim-dev
> >
> >
> >
> > -----Inline Attachment Follows-----
> >
> > _______________________________________________
> > Opensim-dev mailing list
> > Opensim-dev at lists.berlios.de
> > https://lists.berlios.de/mailman/listinfo/opensim-dev
> >
> >
> > ------------------------------
> > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
> > it now.<http://us.rd.yahoo.com/evt=51733/*http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ>
> >
> >
> > _______________________________________________
> > Opensim-dev mailing list
> > Opensim-dev at lists.berlios.de
> > https://lists.berlios.de/mailman/listinfo/opensim-dev
> >
> >
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://opensimulator.org/pipermail/opensim-dev/attachments/20080208/a61c760e/attachment-0001.html>