[Opensim-dev] Fw: Re: Search server DB schema

Tue Feb 5 00:19:26 UTC 2008

Yahoo mail sent the reply to diva's e-mail instead of the whole ML. I've forwarded the original response below.

----- Forwarded Message ----
From: David Wendt JR. <dcrkid at yahoo.com>
To: diva at metaverseink.com
Sent: Monday, February 4, 2008 7:16:50 PM
Subject: Re: [Opensim-dev] Search server DB schema

    I read up on Lucene. It's Java, made by Apache and Apache licensed. Only question, is there a .NET/Mono/C# client library available? One that's BSD compatible? If we have that we can start work on Lucene. As for your comments about relational not being good for search... you might be right. Look how many times Old Search would stop working... they eventually said "screw it" and went with a Google Search Appliance, which works a lot better IMHO.

    Anyway, looking at OpenSim.Framework.Data it may be possible to have our cake and eat it too, provided said cake isn't a lie. Opensim already has a good database abstraction framework that looks like it could also support
 non-SQL/non-relational databases if we so desire. All we would really need to do is write another database plugin for Lucene. At first glance this seems like a very good idea: we can push forward with Search and then write the plugin for the high-performance Lucene search later.

    Devs, what do you think of that?

    UNRELATED NOTE: When we implement search, specifically HTTP Search, we should have an osFunction that lets vendors publish their contents to search.

----- Original Message ----
From: Diva Canto <diva at metaverseink.com>
To: opensim-dev at lists.berlios.de
Sent: Monday, February 4, 2008 6:42:26 PM
Subject: Re: [Opensim-dev] Search server DB schema

Hello 
opensim-developers,

First 
of 
all, 
thanks 
for 
putting 
opensim 
together, 
this 
is 
what 
we 
all 
needed! 
I 
just 
signed 
up 
for 
this 
list, 
so 
apologies 
if 
my 
comments 
are 
out 
of 
place. 
I 
did 
set 
up 
my 
own 
OpenSim, 
and 
it's 
great! 
(in 
spite 
of 
all 
the 
fights 
i've 
been 
having 
with 
mono 
on 
Mac, 
but 
that's 
another 
thread)

Over 
the 
past 
10 
months 
or 
so, 
a 
colleague 
and 
I 
have 
developed 
an 
independent 
search 
engine 
for 
LL's 
SL, 
which 
can 
be 
accessed 
here: 
http://slbrowser.com 
The 
engine 
does 
not 
require 
special 
access 
to 
grid 
databases, 
it 
uses 
bots 
to 
collect 
inworld 
information, 
one 
sim 
at 
a 
time. 
We 
use 
libsecondlife. 
It 
has 
been 
working 
continuously 
for 
the 
past 
6 
months; 
we 
crawl 
the 
grid 
twice 
a 
week 
with 
only 
14 
bots, 
and 
have 
been 
able 
to 
find 
appropriate 
heuristics 
for 
many 
things.

In 
general 
search 
can 
be 
looked 
at 
in 
two 
ways, 
and 
these 
are 
*not* 
incompatible: 
it 
can 
be 
a 
basic 
administrative 
function 
-- 
you 
want 
to 
know 
the 
data 
you 
serve; 
or 
it 
can 
be 
a 
basic 
user 
function 
-- 
you 
want 
people 
to 
be 
able 
find 
things. 
The 
first 
type 
of 
search 
is 
really 
simple: 
stick 
a 
DB, 
and 
you've 
solved 
it. 
This 
works 
fairly 
well 
for 
small 
amounts 
of 
data, 
and 
for 
data 
that 
is 
fairly 
constrained. 
The 
second 
type 
of 
search 
is 
a 
lot 
more 
powerful, 
but 
it's 
not 
so 
simple, 
because 
you 
want 
to 
rank 
the 
huge 
amount 
of 
results 
in 
a 
semantically 
meaningful 
way. 
With 
SLBrowser, 
we've 
followed 
the 
latter. 
Crawling 
the 
live 
sims 
gives 
us 
a 
lot 
of 
useful 
aggregate 
information 
that 
we 
use 
to 
experiment 
with 
ranking 
in 
much 
more 
interesting 
ways. 
We 
don't 
use 
a 
relational 
DB 
on 
the 
backend, 
we 
use 
Lucene.

I 
would 
hate 
that 
OpenSim 
follows 
Linden 
Lab's 
steps 
with 
search 
without 
taking 
advantage 
of 
the 
lessons 
that 
even 
they 
already 
learned 
-- 
that 
relational 
schemas 
are 
not 
appropriate 
for 
modern 
information 
retrieval. 
I'll 
be 
happy 
to 
help 
setting 
up 
this 
basic 
search 
service 
with 
Lucene, 
rather 
than 
with 
a 
relational 
DB. 
Lucene 
is, 
essentially, 
a 
highly 
optimized 
database 
for 
text 
search. 
For 
example, 
issues 
like 
this
" 
I'm 
going 
to 
assume 
name 
== 
varchar(63) 
and 
description 
== 
varchar(127), 
but 
it 
might 
be 
easier 
to 
just 
set 
everything 
to 
varchar(255) 
for 
flexibility."
are 
a 
non-problem 
in 
Lucene 
-- 
you 
can 
use 
as 
little 
or 
as 
much 
text 
as 
you 
want 
in 
a 
field, 
you 
don't 
need 
to 
hard-code 
that.

Can 
I 
help 
with 
plugging 
a 
Lucene-based 
search 
for 
OpenSim, 
please? 
(the 
thought 
of 
having 
a 
relational 
DB 
serving 
text 
search 
makes 
me 
shiver 
:-)
I've 
never 
participated 
in 
an 
Open 
Source 
project 
as 
such, 
so 
I'm 
not 
sure 
how 
the 
process 
is. 
I 
did 
contribute 
to 
OS 
projects 
before 
-- 
aspectj.org, 
co-founder, 
and 
more 
recently 
contributed 
plugins 
to 
XWiki 
with 
one 
of 
my 
students.

Let 
me 
know.

Crista 
Lopes 
/ 
Diva 
Canto
School 
of 
Information 
and 
Computer 
Sciences
University 
of 
California, 
Irvine

_______________________________________________
Opensim-dev 
mailing 
list
Opensim-dev at lists.berlios.de
https://lists.berlios.de/mailman/listinfo/opensim-dev

      Never miss a thing.   Make Yahoo your homepage.

      ____________________________________________________________________________________
Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://opensimulator.org/pipermail/opensim-dev/attachments/20080204/ad228a07/attachment-0001.html>