Knowledge Collector

[Maurits van Rees]

Hi Alain,

Alain Hernandez Lopez, on 2007-05-28:
> I'm developing a product for Plone called information observatory,
> basically the product responsibility is, to establish a connection with
> a previously definite set of information source (urls), to download it
> contents and make it persistent. Now, for the development of this
> project I've created two Plone products, Knowledge Collector and Portal
> Scheduler Tool (tool). The responsibility of Knowledge Collector is to
> establish a connection with information sources, to download it contents
> and make it persistent, and Portal Scheduler Tool has the responsibility
> to check the time, and at midnight it'll execute a set of task.=20

You may want to look at feedfeeder for some inspiration:
http://plone.org/products/feedfeeder

That lets you add a FeedFolder.  A FeedFolder has a list of urls to
atom feeds (rss feeds too I think, but not sure).  You click the
action "Retrieve feed items" and then the new entries of those feeds
get added to the folder.  The clicking can be automated with a
cronjob.  Also zope 2.10 (?) has some clock server that could help,
but I have not looked at that.

> Knowledge Collector is composed by three content types,
> KnowledgeCollector (BaseFolder), InformationSource (BaseContent), and
> DownloadedDocument (BaseContent).

In feedfeeder the InformationSource would be one line of a LinesField
in a FeedFolder.

In your case, InformationSources are stored in a KnowledgeCollector,
right?

And are DownloadedDocuments stored in the KnowledgeCollector too, or
in their own InformationSource?

> Portal Scheduler Tool is a tool that works with thread mechanism, So
> that there are always a main thread running (checking the time) and when
> the right time is, it execute so many thread as task have the Portal
> Scheduler Tool. Basically a task is a Knowledge Collector.

feedfeeder allows you to use wget from a cronjob to do this.  But
having something in Plone itself in which you can do this is nicer
yes.

> How this works? You can create so many Knowledge Collector as you want,
> and when a Knowledge Collector is created automatically a task is set to
> the Portal Scheduler Tool, So that when a tool task is execute, it will
> execute the execute() method of each KnowledgeCollector and then will
> start the process to consult the information sources , to download it
> contents, and finally make it persistent.=20

Sounds clear enough.

> Already I've developed brunt of the project, but I've some troubles
> which I didn't resolve yet, next is the list of problems.
>
> 1.      Once that a task is executed and the information source
> documents are downloaded, is when there to make persistent all this
> documents (DownloadedDocument), my problem is there, when the method
> that create the documents is call, get the next bug.
>
>=20
>
> raise AccessControl_Unauthorized('Cannot create %s' % self.getId())
>
> AccessControl.unauthorized.Unauthorized: Cannot create
> DownloadedDocument

See http://plone.org/documentation/how-to/debug-unauthorized
on how to get a clearer message here.

> I think that's because as this method is execute by a Portal Scheduler
> Tool thread it don't have right permissions to add DownloadedDocuments
> in Knowledge Collector product. Any way I don't know how to resolve
> this.=20
>
> 2.      While Zope is not restarted, KnowledgeCollectos cant be create
> and task are set to the Portal Scheduler Tool perfectly,

Let me rephrase that in light of the rest of the sentence:  before
zope is restarted, KnowledgeCollectors *can* be created.  Correct?

> but once  Zope
> is restarted, and Plone wake up the Portal Scheduler Tool tasks list be
> lost, I've tested making the __listJob  a PersistentList, but this don't
> work to me.=20

PersistentList Should Work (tm).  Can you show us some code?

> 3.      I need that when a KnowledgeCollector is delete a task is delete
> in Portal Scheduler Tool, I've tested making a destructor but it don't
> work, and I didn't found an event handler that resolve this problem.

You could add a method manage_beforeDelete to the KnowledgeCollector
class:

    def manage_beforeDelete(self, item, container):
        # Do something clever here.

See lib/python/OFS/ObjectManager.py.  That is deprecated however.
Instead you can try something with this:

from zope.app.container.contained import ObjectRemovedEvent

-- 
Maurits van Rees | http://maurits.vanrees.org/ [NL]
            Work | http://zestsoftware.nl/
"Do not worry about your difficulties in computers,
 I can assure you mine are still greater."