[PLIP-Advisories] Re: [Plone] #9316: Unify folder implementations
plip-advisories at lists.plone.org
plip-advisories at lists.plone.org
Sat Aug 15 15:26:22 UTC 2009
#9316: Unify folder implementations
----------------------------+-----------------------------------------------
Reporter: smcmahon | Owner: witsch
Type: PLIP | Status: assigned
Priority: n/a | Milestone: 4.0
Component: Infrastructure | Resolution:
Keywords: |
----------------------------+-----------------------------------------------
Old description:
> ,,Copied from [http://plone.org/products/plone/roadmap/191/ PLIP #191] in
> the roadmap:,,
>
> = Unify folder implementations =
>
> ''We currently have "Folder" and "Large Plone Folder" implementations.
> [[br]]
> There should be only one.''
>
> Proposed by::
> Martin Aspeli
> Seconded by::
> Andreas Zeidler
> Proposal type::
> Architecture
> Repository branch::
> [browser:plone.folder], [browser:plone.app.folder]
>
> == Motivation ==
>
> Shipping with two folder types is unnecessary for several reasons:
> - It forces the user to make an a-priori choice about the number of
> objects they plan to put into a folder
> - We ship with the "Large" type disabled by default to avoid UI
> confusion
> - We don't have a proper search-based UI for large folders anyway
>
> Also the standard Folder type stores attributes, and has a single list
> _objects tuple which keeps the list of objects and order. This is prone
> to ConflictErrors and is slower. In simple benchmarks, a BTree-based
> folder performs orders of magnitude faster than a basic folder.
>
> == Proposal ==
>
> Have a single folder implementation.
> - The internal storage is BTrees
> - It still supports ordering, by storing a separate sort order list/tree
> - It has at least two views - one search-based for "large" folders, one
> batch-based for "small" folders. This is either just a "display" menu
> choice, or a choice in the object's schema. Explicit sorting may be
> turned off for "large" folders.
>
> == Implementation ==
>
> The package `plone.folder` in the Plone SVN provides a base
> implementation of a folder base class, which is not Archetypes specific,
> based on BTreeFolder2, but adding ordering. The exact ordering
> implementation is left up to an adapter, with a default providing
> explicit ordering. This allows other implementations, such as auto-
> sorting based on some key.
>
> The diagram below shows the folderish base- and mix-in classes used in
> OFS, CMF, Archetypes and Plone. Count 'em:
>
> [[Image(http://plone.org/products/plone/roadmap/191/Folder%20mess.png)]]
>
> An initial investigation suggests that this could be made a base class
> for BaseBTreeFolder in Archetypes. This is backwards compatible, since
> the new type simply replaces the rather simple CMFBTreeFolder.
>
> We'd then need to make this folder type the default, and ensure that
> there's adequate migration (either in that existing instances continue to
> work but new folders have this feature, maybe with an optional "migrate"
> action, or through some mass migration).
>
> The Container type in plone.app.content does not use BTrees, nor does it
> support ordering. Rather than changing this class and providing
> migration, we could add a new type, e.g. called OrderedContainer or just
> Folder (to signify closer resemblance to the standard folder behaviour)
> that mixes in the new class instead of PortalFolder.
>
> == Deliverables ==
>
> - Improved Products.ATContentTypes.content.ATBTreeFolder that uses
> BTrees and supports ordering.
> - New class plone.app.content.container.Folder (?) that uses BTrees and
> supports ordering.
> - An out-of-the-box Folder type that is BTree-based and supports large
> content sets as well as ordering.
> - The ability to switch to a folder view/behaviour that's optimised for
> "large" content sets.
> - Migration/graceful fallback for all existing sites.
>
> == Risks ==
>
> - There may be non-trivial migration involved if existing instances
> should be changed.
> - This vision makes the existing BaseBTreeFolder in Archetypes orderable
> (though in some UI configurations you may not see the ordering, to
> prevent the user invoking slow operations), which may be unexpected
>
> == Progress ==
>
> Complete:
>
> - Create a base folder implementation - OrderedBTreeFolderBase
> - Hook this into BaseBTreeFolder in Archetypes
> - Fix CMFPlone so that it detects the order support based on a Zope 3
> interface rather than a Zope 2 one
> - Test that order support works on the new folder
> - Switch FTIs and portal_type's around so that the default "Folder"
> portal_type is a btree-based folder
> - Enable next/previous navigation for this folder type
> - Enable photo-album support for this folder type
> - Enable archiving support for this folder type
> - Enable ordering policy to be specified by adapter
>
> To-do - ATContentTypes:
>
> - Fix remaining ATCT and CMFPlone test failures (mainly around WebDAV)
> - Write migrations for FTI and portal_type changes
> - Provide optional migration to change types in-place
>
> To-do - plone.app.content:
>
> - Create new Folder base class, using the new mixin, with tests
>
> == Benchmark results ==
>
> The following two graphs show some simple benchmarks using the various
> types of folders (regular = ATFolder, large = ATBTreeFolder, ordered =
> ATBTreeFolder w/ order support) with 50 and 500 items, respectively. The
> tests conducted where creating content, retrieving a batch of items from
> the folder, getting all items from the folder as well as random accessing
> the items. The benchmark tests used can be found in the
> [browser:plone.folder/trunk/src/plone/folder/tests/ tests/] folder.
>
> [[Image(http://plone.org/products/plone/roadmap/191/folder_benchmarks_50_items.png)]]
> [[Image(http://plone.org/products/plone/roadmap/191/folder_benchmarks_500_items.png)]]
>
> A more realistic scenario was provided by some benchmarks based on
> [http://jakarta.apache.org/jmeter/ JMeter] (available in
> [browser:plone.folder/jmeter/ jmeter]):
>
> [[Image(http://plone.org/products/plone/roadmap/191/jmeter_performance_tests.png)]]
>
> The graphs show the time it took to create a folder with 500 news items
> both using a standard folder and the new btree-based folder with order
> support. Also, and that's probably the most interesting figure so far,
> the time for repeatedly browsing the "folder contents" view is decreased
> to slightly less than 75% by using the btree folder.
>
> Another interesting thing to note is that the "content creation" tests
> show that btree-based folders are faster than the currently used
> implementation (from ATFolder) even for very small numbers of contained
> objects. The tests still run faster when adding 500, 50, 10, 5, 2 or
> even just a single object to the folder.
>
> == Participants ==
>
> - Martin Aspeli (IRC nickname: <optilude>)
> - Andreas Zeidler (IRC nickname: <witsch>)
> - Martijn Pieters (IRC nickname: <MJ>)
> - Alec Mitchell (IRC nickname: <alecm>)
New description:
,,Copied from [http://plone.org/products/plone/roadmap/191/ PLIP #191] in
the roadmap:,,
= Unify folder implementations =
''We currently have "Folder" and "Large Plone Folder" implementations.
[[br]]
There should be only one.''
Proposed by::
Martin Aspeli
Seconded by::
Andreas Zeidler
Proposal type::
Architecture
Repository branch::
[browser:plone.folder], [browser:plone.app.folder]
== Motivation ==
Shipping with two folder types is unnecessary for several reasons:
- It forces the user to make an a-priori choice about the number of
objects they plan to put into a folder
- We ship with the "Large" type disabled by default to avoid UI confusion
- We don't have a proper search-based UI for large folders anyway
Also the standard Folder type stores attributes, and has a single list
_objects tuple which keeps the list of objects and order. This is prone to
ConflictErrors and is slower. In simple benchmarks, a BTree-based folder
performs orders of magnitude faster than a basic folder.
== Proposal ==
Have a single folder implementation.
- The internal storage is BTrees
- It still supports ordering, by storing a separate sort order list/tree
- It has at least two views - one search-based for "large" folders, one
batch-based for "small" folders. This is either just a "display" menu
choice, or a choice in the object's schema. Explicit sorting may be turned
off for "large" folders.
== Implementation ==
The package `plone.folder` in the Plone SVN provides a base implementation
of a folder base class, which is not Archetypes specific, based on
BTreeFolder2, but adding ordering. The exact ordering implementation is
left up to an adapter, with a default providing explicit ordering. This
allows other implementations, such as auto-sorting based on some key.
The diagram below shows the folderish base- and mix-in classes used in
OFS, CMF, Archetypes and Plone. Count 'em:
[[Image(http://plone.org/products/plone/roadmap/191/Folder%20mess.png)]]
The new base class from `plone.folder`, i.e. `OrderedBTreeFolderBase`, is
used by `plone.app.folder` to provide two folderish classes, one targeted
to `Archetypes` (`BaseBTreeFolder`) as well as one for `ATContentTypes`
(`ATFolder`). Both add relatively little extra code and setup in order to
make them fully compatible with the original, to be replaced classes.
The package also provides a `GenericSetup` profile replacing the standard
"Folder" content type with the new one. In-place migration will convert
the internal data-structures when upgrading to use the new folders. Such
migration code (including thorough tests) already exists in a project-
specific package, and just needs to be moved into `plone.folder` itself.
The migration runs relatively fast |---| in several performance tests
about 13,000 folders holding some 200,000 items in total were migrated in
about 5 minutes.
.. |---| unicode:: U+2014 .. em dash
The Container type in `plone.app.content` does not use BTrees, nor does it
support ordering. Rather than changing this class and providing migration,
we could add a new type, e.g. called `OrderedContainer` or just `Folder`
(to signify closer resemblance to the standard folder behaviour) that
mixes in the new class instead of `PortalFolder`.
In addition, `plone.folder` also provides an ordering adapter, which only
considers certain content to be orderable (implemented via marker
interfaces). This allows for ordering of content that requires it, for
example navigational items, without any added performance penalties if
those share a folder with other, non-orderable content in large
quantities.
== Deliverables ==
- New folderish base classes that are BTree-based and support large
content sets as well as ordering.
- Improved `Products.ATContentTypes.content.ATFolder` and
`Products.ATContentTypes.content.ATBTreeFolder` versions which use BTrees
and supports ordering.
- New class plone.app.content.container.Folder (?) that uses BTrees and
supports ordering.
- The ability to switch to a folder view/behaviour that's optimised for
"large" content sets.
- In-place Migration for all existing sites.
== Risks ==
- There may be migration issues involved for derived folderish types with
additional data-structures regarding containment and/or ordering.
- This vision makes the existing `BaseBTreeFolder` in Archetypes
orderable (though in some UI configurations you may not see the ordering,
to prevent the user invoking slow operations), which may be unexpected.
== Progress ==
Complete:
- Create a base folder implementation - OrderedBTreeFolderBase
- Hook this into BaseBTreeFolder in Archetypes
- Fix CMFPlone so that it detects the order support based on a Zope 3
interface rather than a Zope 2 one
- Test that order support works on the new folder
- Update FTIs and portal_type's so that the default "Folder" portal_type
is a btree-based folder
- Enable next/previous navigation for this folder type
- Enable photo-album support for this folder type
- Enable archiving support for this folder type
- Enable ordering policy to be specified by adapter
Releases:
- 1.0b3 of `plone.folder` has been released in May and is in Dexterity.
- 1.0a1 of `plone.app.folder` has been released in May in order to factor
out all code specific to `ATContentTypes`
- Both releases have been heavily tested in a large-scale Jarn project
and will go into production next week.
To-do - ATContentTypes:
- Fix remaining ATCT and CMFPlone test failures (mainly around WebDAV)
- Merge in-place migration to new folder types
- Write migrations for FTI and portal_type changes
- Ensure compatibility with Plone 4.0, i.e. Python 2.6 & Zope 2.12
To-do - plone.app.content:
- Create new Folder base class, using the new mixin, with tests
== Benchmark results ==
The following two graphs show some simple benchmarks using the various
types of folders (regular = ATFolder, large = ATBTreeFolder, ordered =
ATBTreeFolder w/ order support) with 50 and 500 items, respectively. The
tests conducted where creating content, retrieving a batch of items from
the folder, getting all items from the folder as well as random accessing
the items. The benchmark tests used can be found in the
[browser:plone.folder/trunk/src/plone/folder/tests/ tests/] folder.
[[Image(http://plone.org/products/plone/roadmap/191/folder_benchmarks_50_items.png)]]
[[Image(http://plone.org/products/plone/roadmap/191/folder_benchmarks_500_items.png)]]
A more realistic scenario was provided by some benchmarks based on
[http://jakarta.apache.org/jmeter/ JMeter] (available in
[browser:plone.folder/jmeter/ jmeter]):
[[Image(http://plone.org/products/plone/roadmap/191/jmeter_performance_tests.png)]]
The graphs show the time it took to create a folder with 500 news items
both using a standard folder and the new btree-based folder with order
support. Also, and that's probably the most interesting figure so far,
the time for repeatedly browsing the "folder contents" view is decreased
to slightly less than 75% by using the btree folder.
Another interesting thing to note is that the "content creation" tests
show that btree-based folders are faster than the currently used
implementation (from ATFolder) even for very small numbers of contained
objects. The tests still run faster when adding 500, 50, 10, 5, 2 or even
just a single object to the folder.
== Participants ==
- Martin Aspeli (IRC nickname: <optilude>)
- Andreas Zeidler (IRC nickname: <witsch>)
- Martijn Pieters (IRC nickname: <MJ>)
- Alec Mitchell (IRC nickname: <alecm>)
--
Comment(by witsch):
update ticket to reflect the current status
--
Ticket URL: <http://dev.plone.org/plone/ticket/9316#comment:20>
Plone <http://plone.org>
Plone Content Management System
More information about the PLIP-Advisories
mailing list