[Framework-Team] Re: PLIP 177 - Content indexing

[Andrzej (nickSamotnik)]

Kapil Thangavelu <k_vertigo at ...> writes:

> 
> > Kapil Thangavelu wrote:
> >> On Mon, 11 Dec 2006 08:55:07 -0500, Rocky Burt
> >> <rocky at ...> wrote:
> >>
> >>> On Mon, 2006-11-12 at 13:43 +0000, Martin Aspeli wrote:

> >>> Personally I think this plip would be awesome to get in place.  But  
> >>> with
> >>> one small stipulation.  I would adding a configlet someplace or some
> >>> config option somewhere that lets you toggle this behaviour on/off (on  
> >>> a
> >>> functional level) and have it off by default.  That way we don't have  
> >>> to
> >>> worry about the performance implications by default.  Then for Plone  
> >>> 3.5
> >>> we could consider having it activated by default.
> >>
> >>
> >> +1, hmm.. instead of field changes, maybe its better to just restructure
> >> the catalog itself, to use real adapters to index content, with a
> >> default implementation respecting this setting. more work, but more
> >> flexible for repurposing on other use cases, probably 3.5 material.
> >
> > Why not include TXNG3?
> 
> txng3 is a huge package, its not just a plugin index, its basically its  
> own catalog infrastructure, with lots of code, including c extensions,  
> with one maintainer, afaik. 98% of the people i bet install it for one  
> reason, namely the focus of this plip, indexing common office file types,  
> and all its extra complexity, features, and options ignored. for this  
> particular purpose, under the hood txng3 is utilizing the same machinery,  
> so its best i think to just give the functionality that most users already  
> want, is already in the codebase, via just exposing the functionality, as  
> opposed to including an entirely new framework that needs to be supported  
> and maintained.
> 
> cheers,
> 
> kapil
> 

Great PLIP, Kapil, but much more is needed - the order in all indexing/searching
infrastructure! It is necessary because effective searching becomes critical
issue for many companies, so as for administrators of Plone applied for various
tasks, including document management/repositories for example.

First, a lot of standardisation is needed. Some examples of one issue only, the
pipeline:

* Pipeline in CMFPlone is loaded using XML file. In other products, like
TextIndexNG3 or ExternalSiteCatalog not,

* It is impossible to turn Casefolding (Case Normalization) off under Windows;
under Linux it is possible at the cost of Unicode options,

* When using TextIndexNG3 for indexing ExternalSiteCatalog the HTML code is not
removed; this indicates a serius archietcture error, probably in
ExternalSiteCatalog. There should be a standard of componentisation or
pluggization of the pipeline.

* There is not unified way of loading and configuring lexicons, not clear
division of processing tasks between program and lexicon, and not unified
location of lexicon loading/initiation/configuring; for example: where is the
lexicon of TextIndexNG3?

Second, the indexing/searching infrastructure should be treated as the product.
So, some minimum requirements should be established for examples, documentation
etc. For example, there is example of user interface (query form and results'
listing) in ExternalSiteCatalog, a chapter in Zope Book for ZTCIndex for ZMI
interface, no any documentation for TextIndexNG3. Please take into account that
Plone administrator is not necessarily the qualified developer.

Third, the architecture of indexing/searching infrastructure is a big subject
for consideration. The great advantage of TextIndexNG3 is a pluggable
architecture which is absolutely necessary for the users without developer's
background. In practice, however, it is the illusion, to some extent, because of
very complex construction of this great product. Pluggization is indispensable
for elasticity, but requires good integrating mechanisms, so it is obviously not
easy to find the compromise between independence of the plug and its dependency
on the middle strata. Big issue.

It is a voice of non-developer, so excuse possible incorrectnesses.

Samotnik