Can you please give an overview of Spear?--sounds cool, but not sure what it is/does...is it a custom content creation utility without the overhead of Archetypes?<br><br><div class="gmail_quote">On Wed, Jan 7, 2009 at 3:26 AM, Souheil CHELFOUH <span dir="ltr"><<a href="mailto:trollfot@gmail.com">trollfot@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">You can also check the project i'm working on, Spear, for lightweight<br>
and already working content type mini framework<br>
<a href="http://tracker.trollfot.org/browser/projects/spear.example/trunk/spear/example" target="_blank">http://tracker.trollfot.org/browser/projects/spear.example/trunk/spear/example</a><br>
<br>
2009/1/5 Mikko Ohtamaa <<a href="mailto:mikko%2Bplone@redinnovation.com">mikko+plone@redinnovation.com</a>>:<br>
<div><div></div><div class="Wj3C7c">><br>
><br>
><br>
> Andreas Jung wrote:<br>
>> On 05.01.2009 14:04 Uhr, Martin Aspeli wrote:<br>
>>> Mikko Ohtamaa wrote:<br>
>>>> Hi,<br>
>>>><br>
>>>> We are facing a problem where we need to store 270 fields per item. The<br>
>>>> fields are laboratory measurements of a patient - 40 measurement<br>
>>>> values for<br>
>>>> 7 timepoint. The fields need to be accessed per timepoint, per<br>
>>>> measurement<br>
>>>> and all fields for one patient once. There will be over 10000 patients,<br>
>>>> distributed under different hospital items (tree-like, for permission<br>
>>>> reasons). Data is not accessed for two patients at once, so we don't<br>
>>>> need to<br>
>>>> scale the catalog.<br>
>>>><br>
>>>> So I am curious about how we make Plone scale well for this scenario.<br>
>>>><br>
>>>> - The overhead of a field in AT schema? Should we use normal storage<br>
>>>> backend<br>
>>>> (Python object value) or can we compress or field values into<br>
>>>> list/dict to<br>
>>>> make it faster using a custom storage backend.<br>
>>>><br>
>>>> - The wake up overhead of AT object? Should we distribute our fields to<br>
>>>> several ZODB objects e.g. per timepoint, or just stick all values to one<br>
>>>> ZODB objects. All fields per patient are needed on some views once.<br>
>>>><br>
>>>> - One big Zope objects vs. few smaller Zope objects?<br>
>>> I wouldn't store this in the ZODB, at least not only in the ZODB. Values<br>
>>> like this are better stored in an RDBMS, modelled e.g. with a 40-column<br>
>>> table (ick) used 7 times (one for each time point) for each patient.<br>
>><br>
>> This is possibly ovehead - especially with collective.tin. Storing the<br>
>> data within some BTree datastructure should scale fine and requires a<br>
>> lot less effort than using collective tin.<br>
><br>
>>True. I was assuming he would need more complex data querying, though.<br>
>>Anything that requires non-trivial joins is probably better served by an<br>
>>RDMBS backend and SQL.<br>
><br>
> First thank you for everyone for very insightful replies!<br>
><br>
> I can fill in few gaps and have few more questions:<br>
><br>
> - Archetypes is used as a generic framework to provide function structures,<br>
> but all view and edit pages will be customized in any case - we are not<br>
> going to dump the whole schema on the edit page once - the generated HTML<br>
> and widget code will be optimized later on.<br>
><br>
> - We will use ore.contentmirror to mirror the data to SQL database for data<br>
> mining. We don't need real-time mirroring or real time queries, thus the<br>
> cataloging is not an issue<br>
><br>
> - Objects are mostly write-once - data shouldn't need to be changed unless<br>
> there has been an input error<br>
><br>
> - We are probably going to split data to objects based by timepoints, as<br>
> suggested<br>
><br>
> - *Is it still desirable to have 7 smaller ZODB objects than one big<br>
> object*? What is "the breaking point" of schema when AttributeStorage falls<br>
> apart? We need to query 7 ZODB objects to the patient main view to render<br>
> the table containing the patient summary data.<br>
><br>
> - What should we keep in mind if we indent to replace AttributeStorage with<br>
> a custom BTreeStorage?<br>
><br>
> Cheers,<br>
> Mikko<br>
> --<br>
> View this message in context: <a href="http://n2.nabble.com/The-most-efficient-way-to-store-270-AT-fields--tp2112645p2112902.html" target="_blank">http://n2.nabble.com/The-most-efficient-way-to-store-270-AT-fields--tp2112645p2112902.html</a><br>
> Sent from the Product Developers mailing list archive at Nabble.com.<br>
><br>
><br>
> _______________________________________________<br>
> Product-Developers mailing list<br>
> <a href="mailto:Product-Developers@lists.plone.org">Product-Developers@lists.plone.org</a><br>
> <a href="http://lists.plone.org/mailman/listinfo/product-developers" target="_blank">http://lists.plone.org/mailman/listinfo/product-developers</a><br>
><br>
<br>
_______________________________________________<br>
Product-Developers mailing list<br>
<a href="mailto:Product-Developers@lists.plone.org">Product-Developers@lists.plone.org</a><br>
<a href="http://lists.plone.org/mailman/listinfo/product-developers" target="_blank">http://lists.plone.org/mailman/listinfo/product-developers</a><br>
</div></div></blockquote></div><br>