We are monitoring a few applications (JAVA and .NET) which communicate with each other. We had enabled cross process transaction tracing, however the correlated transaction traces do not seem to be in correct order.
The first application that calls the second and subsequent application, shows up at the bottom. The subsequent calls show up at the top.
For example, the flow is suppose to be like this:
.NET server App Pool 1 -> .NET Server App Pool 2 -> Java server
WebAPI -> CG ->newStars
However, in Historical Query Viewer sorted by Timestamp, traces were seen in an opposite sequence, where trace from newStars application was seen first, followed by CG then WebAPI.
Transaction components in Trace View also does not resemble an "upside-down wedding cake":
There is a time difference of a few seconds between the monitored application/agent servers.
Synchronizing the server time for all the monitored applications resolved the problem. Traces show up in the correct sequence since then.