[#MULE-774] Overhaul QueuePersistenceStrategies

Mule

Overhaul QueuePersistenceStrategies

Created: 17/Apr/06 10:49 AM Updated: 10/Dec/07 08:15 AM

Component/s:

Core: Queues (SEDA) / Persistence

Affects Version/s:

1.3-rc2

Fix Version/s:

None

Time Tracking:

Not Specified

Issue Links:

Related

This issue relates to:

~~MULE-703~~ Journalling test cases fail sporadically

MULE-1330 HierarchicalFilePersistenceStrategy (like Squid)

This issue relates to:

MULE-2158 Getting OutOfMemoryError while running stress test in MulticastRouterMule2136TestCase

~~MULE-1300~~ IdempotentReceiver has guaranteed memory leak

~~MULE-891~~ Plan roadmap to 2.0 via 1.3.x, 1.4.x

Labels:

Description

« Hide

The QueuePersistenceStrategy implementations can benefit greatly from small but valuable improvements:

MemoryPersistenceStrategy (the default?) should use per-queue stores (a Map of Maps) to reduce contention; using ConcurrentHashMaps will probably do wonders with multiple threads.

CachingPersistenceStrategy caches written values but does not read them - load() always reads-through to the backend? Luckily remove() does the right thing, otherwise it would be a really big memory hog (the caching ReferenceMap keeps SoftReferences, which is a bit questionable in itself - customizable expiry and fixed-size might be preferrable)

FilePersistenceStrategy could use the queue name for subdirectories; this would greatly reduce the contention on the root directory (file creation/deletion are necessarily atomic operations in most file systems and as such great contention points). With the help of a few reader threads performance will probably skyrocket as well, though there should be a limit to the number of workers, otherwise drive head trashing will make this actually slower. Need to write a benchmark with a crappy file system, say NTFS.

The JournalPersistenceStrategy really, really needs to be rewritten or at least fixed ASAP (~~MULE-703~~). Currently it uses ActiveIO with its own backend but it would probably be better to directly use Howl (http://howl.objectweb.org). ActiveIO can use Howl as backend (acting as a very thin wrapper) but as far as I can see from the source this is currently incomplete. A direct implementation of Howl without any additional abstraction sugar will likely offer better fault resilience and performance. Another option - most likely easier to implement - might be to use BerkeleyDB Java edition (http://www.sleepycat.com/products/bdbje.html).

All

Comments

Work Log

Change History

Transitions

FishEye

Sort Order:

[ Permlink | « Hide ]

Holger Hoffstaette added a comment - 28/Apr/06 08:00 AM

re: FilePersistenceStrategy & subdirectories - turns out it aready does exactly that, for obvious reasons.

[ Permlink | « Hide ]

Holger Hoffstaette added a comment - 10/Aug/06 10:35 PM

The more I think about this it seems to me that CachingPersistenceStrategy is redundant since all it does is get in the way without benefit? When an event is consumed it must read from the backend; when it is written it must write through. The only possible benefit might be if it acted like a small read-ahead buffer by reading/writing batches. However, that would mess with the PersistenceStrategy's consistency on failure.