
Description of a problem with synchronous replication
=====================================================

// $Id: SynchronousReplicationWoes.txt,v 1.1.1.1 2005/03/31 10:10:39 belaban Exp $

Problem description
-------------------

- 2 processes, A (TreeCacheAopView) and B (beanshell). Synchronous replication
- B starts a TX, makes a change, and commits TX
- B's replication interceptor calls PREPARE on A with GTX
- A starts a new local TX, and executes the methods shipped with PREPARE
- As a result of one of the method invocations, a TreeCacheListener does a get() on the cache
- The get() causes the ReplicationInterceptor to register for TX completion with the TxManager
- A's ReplicationInterceptor calls COMMIT on B
- COMMIT on B causes the TxManager to notify the registered ReplicationInterceptor (before/afterCompletion())
- The ReplicationInterceptor sees it has modifications associated with the local TX and starts running
  the PREPARE, this time on A ! This essentially tries to apply the changes A originally sent to *B* back to A !
  We have a never-ending cycle (A will re-replicate to B etc etc) !


Solution
--------

- Tag the TransactionEntry associated with the local TX on A, if the local TX was created as a result of receiving
  a PREPARE, with fake_tx=true
- When a ReplicationInterceptor attempts to register for TX completion, it checks fake_tx: if true, it will
  *not* register. Same for beforeCompletion() (should not happen though, just a 2nd line of defense): if fake_tx
  for the TransactionEntry associated with the local TX is true, beforeCompletion() returns immediately
- This scheme will not result in cycles


Issues
------

- With this scheme, if a callback executed as result of method invocations shipped with PREPARE (e.g. on A)
  *modifies* the cache itself (e.g. put() in a nodeModified() callback), the resulting modification will *not*
  be replicated to B because the ReplicationInterceptor does not register for TX completion. If the TX is rolled
  back, the changed made in nodeModified() (e.g. the put()) are rolled back
- We may change these semantics further down the road, but we suggest not to modify the cache in TreeCacheListener
  callbacks anyway...


  UPDATE !!
  =========

- This was solved differently: we keep track of the modifications on reception of PREPARE, and - when the local TX
  commits - we *remove* all modifications sent with PREPARE from the modifications associated with the local TX
  in the TransactionTable; this way, we only replicate if some callback made some additonal modifications to
  the cache
- The consequence is that now *all* callback are allowed to make modifications to the cache !