[Product-Developers] Re: The most efficient way to store 270 AT fields?

Andreas Jung lists at zopyx.com
Mon Jan 5 13:15:16 UTC 2009


On 05.01.2009 14:04 Uhr, Martin Aspeli wrote:
> Mikko Ohtamaa wrote:
>> Hi,
>>
>> We are facing a problem where we need to store 270 fields per item. The
>> fields are laboratory measurements of a patient - 40 measurement
>> values for
>> 7 timepoint. The fields need to be accessed per timepoint, per
>> measurement
>> and all fields for one patient once. There will be over 10000 patients,
>> distributed under different hospital items (tree-like, for permission
>> reasons). Data is not accessed for two patients at once, so we don't
>> need to
>> scale the catalog.
>>
>> So I am curious about how we make Plone scale well for this scenario.
>>
>> - The overhead of a field in AT schema? Should we use normal storage
>> backend
>> (Python object value) or can we compress or field values into
>> list/dict to
>> make it faster using a custom storage backend.
>>
>> - The wake up overhead of AT object? Should we distribute our fields to
>> several ZODB objects e.g. per timepoint, or just stick all values to one
>> ZODB objects. All fields per patient are needed on some views once.
>>
>> - One big Zope objects vs. few smaller Zope objects?
>
> I wouldn't store this in the ZODB, at least not only in the ZODB. Values
> like this are better stored in an RDBMS, modelled e.g. with a 40-column
> table (ick) used 7 times (one for each time point) for each patient.

This is possibly ovehead - especially with collective.tin. Storing the
data within some BTree datastructure should scale fine and requires a 
lot less effort than using collective tin.

Andreas--
ZOPYX Ltd. & Co. KG - Charlottenstr. 37/1 - 72070 Tübingen - Germany
Web: www.zopyx.com - Email: info at zopyx.com - Phone +49 - 7071 - 793376
Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535
Geschäftsführer/Gesellschafter: ZOPYX Limited, Birmingham, UK
------------------------------------------------------------------------
E-Publishing, Python, Zope & Plone development, Consulting

-------------- next part --------------
A non-text attachment was scrubbed...
Name: lists.vcf
Type: text/x-vcard
Size: 316 bytes
Desc: not available
URL: <http://lists.plone.org/pipermail/plone-product-developers/attachments/20090105/c1f8083d/attachment.vcf>


More information about the Product-Developers mailing list