[Product-Developers] Re: The most efficient way to store 270 AT fields?

Martin Aspeli optilude at gmx.net
Mon Jan 5 13:22:30 UTC 2009


Andreas Jung wrote:
> On 05.01.2009 14:04 Uhr, Martin Aspeli wrote:
>> Mikko Ohtamaa wrote:
>>> Hi,
>>>
>>> We are facing a problem where we need to store 270 fields per item. The
>>> fields are laboratory measurements of a patient - 40 measurement
>>> values for
>>> 7 timepoint. The fields need to be accessed per timepoint, per
>>> measurement
>>> and all fields for one patient once. There will be over 10000 patients,
>>> distributed under different hospital items (tree-like, for permission
>>> reasons). Data is not accessed for two patients at once, so we don't
>>> need to
>>> scale the catalog.
>>>
>>> So I am curious about how we make Plone scale well for this scenario.
>>>
>>> - The overhead of a field in AT schema? Should we use normal storage
>>> backend
>>> (Python object value) or can we compress or field values into
>>> list/dict to
>>> make it faster using a custom storage backend.
>>>
>>> - The wake up overhead of AT object? Should we distribute our fields to
>>> several ZODB objects e.g. per timepoint, or just stick all values to one
>>> ZODB objects. All fields per patient are needed on some views once.
>>>
>>> - One big Zope objects vs. few smaller Zope objects?
>> I wouldn't store this in the ZODB, at least not only in the ZODB. Values
>> like this are better stored in an RDBMS, modelled e.g. with a 40-column
>> table (ick) used 7 times (one for each time point) for each patient.
> 
> This is possibly ovehead - especially with collective.tin. Storing the
> data within some BTree datastructure should scale fine and requires a 
> lot less effort than using collective tin.

True. I was assuming he would need more complex data querying, though. 
Anything that requires non-trivial joins is probably better served by an 
RDMBS backend and SQL.

Martin

-- 
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book





More information about the Product-Developers mailing list