History | Log In     View a printable version of the current page.  
Issue Details (XML | Word | Printable)

Key: MULE-1773
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Critical Critical
Assignee: Holger Hoffstaette
Reporter: Marie Claire Rizzo
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
Mule

Thread Interference in the FileMessageDispatcher

Created: 21/May/07 05:30 AM   Updated: 23/Jan/08 06:36 AM
Component/s: Transport: File
Affects Version/s: 1.4.0
Fix Version/s: 2.0.0-RC2, 1.4.4

Time Tracking:
Not Specified

Environment: Mule 1.4
Issue Links:
Related
 

Labels:
User impact: High


 Description  « Hide
The scenario is the following:

A number of messages need to be written to a specified file. The messages arrive fine up to the FileMessageDispatcher but once they are written, some messages are missing from the file. Note that not the same amount of messages are lost. Some times all messages arrive fine. This problem probably occurs due to thread interference, with the output of one thread being overwritten by that of another.

Synchronizing the doDispatch() method seems to solve the problem. The code modification suggested is as seen below:

synchronized (mutex)
{
FileOutputStream fos = (FileOutputStream)connector.getOutputStream(event.getEndpoint(), message);
if (event.getMessage().getStringProperty(FileConnector.PROPERTY_FILENAME, null) == null)

{ event.getMessage().setStringProperty(FileConnector.PROPERTY_FILENAME, message.getStringProperty(FileConnector.PROPERTY_FILENAME, "")); }

try

{ fos.write(buf); }

finally

{ fos.close(); }

}

The mutex is declared as follows:

private final static Object mutex = new Object();

Any thoughts on this?



 All   Comments   Work Log   Change History   Transitions   FishEye      Sort Order: Ascending order - Click to sort in descending order
Holger Hoffstaette - 21/May/07 05:49 AM
I was about to say that the mutex would have no effect but then saw that you declared it as static. That was correct but still unnecessary
In 1.4 all dispatcher actions are declared as unsynchronized for the dispatcher instance and always guaranteed to be performed by a single thread only, so only multiple concurrent requests to the same endpoint can conflict since they are done by multiple instances. The file transport is one of the modules that so far were not yet reviewed for the new transport architecture (see MULE-1436).
In the meantime check out AbstractConnector.get/setMaxDispatchersActive which was made for exactly the purpose of limiting the number of concurrent dispatchers. The "real" solution is to have a mutex (lock manager) by endpoint (directory path/file).
So as short-term fix configuring or hard-coding the file connector to return a constant 1 for getMaxDispatchersActive should have exactly the same effect as your mutex.

Holger Hoffstaette - 14/Aug/07 07:57 AM
Marie, can we have a test case for this? It's important to know what exactly it means that some messages are missing from the file (which should never happen). Are exceptions silently swallowed? Are the contents mixed up? Also it's important to see that the order of the file contents is not guaranteed with async flows since threads can overtake each other.

Dirk Olmes - 11/Sep/07 12:02 PM
Marie, see if you can get this fixed for the 1.4.3 release

Marie Claire Rizzo - 18/Sep/07 07:49 AM
I can't create a proper test case for this since it only occurs on very large amounts of data being processed concurrently. However, there are no exceptions being thrown, just one batch of messages overwriting others. The only indication of the messages being lost is in the number of messages being written to the output file which is less than that expected.

Holger Hoffstaette - 18/Sep/07 08:24 AM
Marie, Dirk & I will write a test case and research the broader problem of file locking in general. Once we can rreliably reproduce the problem we'll provide at least a short-term fix.

Marie Claire Rizzo - 18/Sep/07 08:25 AM
Ah, sorry. I will retest to see whether the behaviour has been fixed with the changes implemented in MULE-966.

Holger Hoffstaette - 18/Sep/07 08:38 AM
It is unlikely that the problem is fixed - I can easily think of a case where sending data to a file concurrently (via dispatch()) is mangled, especially in append mode. When there was only a single dispatcher for an endpoint, it would have been enough to make the doDispatch() synchronized; now there are multiple (pooled) and they are not coordinated. That is why i suggsted to limit their number to 1 as quick fix.
I guess in a way MULE-966 has created the bug :-}

Holger Hoffstaette - 20/Sep/07 09:47 AM
moving to Dirk (and me)

Holger Hoffstaette - 23/Jan/08 06:36 AM
I have limited maxDispatchersActive to 1 in:

http://fisheye.codehaus.org/changelog/mule/?cs=10483
http://fisheye.codehaus.org/changelog/mule/?cs=10484

The default value (set in the constructor) and overridden setter can be removed once some kind of LockManager exists. This will of course not change the behaviour - it's just not possible to append to the same file with multiple writers without some kind of locking.