Abstract We can know the past by accurately remembering events we experienced. Yet whether a memory accurately represents a past event depends on whether the memory successfully refers to the event. How do memories refer to their objects? One answer to this question falls neatly out of traditional causal theories of memory: reference is secured by a causal link, sustained through a memory trace. However, recent advances in memory science suggest that remembering is an inherently constructive rather than preservative process. Motivated by these empirical findings, the new philosophical paradigm treats remembering as a species of constructive imagining. This paper raises a novel challenge for the most influential representative of the constructivist paradigm: simulation theory. The challenge is to explain how remembering, so understood, could secure the right referential link to past events. It argues that simulation is unable to perform the reference-fixing role previously assigned to memory traces. In dropping the causal condition on remembering, simulationism has no viable route to explaining memory-based reference to, and therefore knowledge of, the past.