Support time-based event store queries

[JBetz]

There's currently no way to do time-based queries on the event store since timestamps aren't recorded unless you add them to your event types explicitly. They should be added to the StoredEvent type, or a new type along with a new EventStore should be created.

One of the most important use cases for event sourcing is to support time travel, and if the event store doesn't have any notion of time, this is impossible. I'm interested in hearing the reasons for not including it, but given the goals of event sourcing, it seems too important to exclude.

[jdreaver]

Indeed, time travel is important, but that is done by traversing a stream of ordered events, not necessarily by using timestamps. In eventful, you get stream ordering with EventVersion and global ordering with SequenceNumber.

You are free to add your own timestamps to an event store! In particular, this is really easy with the SQL event stores: just add a created_at column to your table with a default of now(). eventful doesn't even have to know about it! eventful was designed to have stores be extensible. StoredEvent has the fields it has because that is all eventful needs to know about. Your actual event store implementation can have more metadata attached.

You could also add timestamps to events explicitly and deal with generating them yourself. Just wrap your event type in a data type with UTCTime attached. Again, eventful doesn't need to know about it because it won't use that information.

[JBetz]

Indeed, time travel is important, but that is done by traversing a stream of ordered events, not necessarily by using timestamps.

Fair point, I just prefer to refer to time explicitly when doing so. :)

You are free to add your own timestamps to an event store! In particular, this is really easy with the SQL event stores: just add a created_at column to your table with a default of now(). eventful doesn't even have to know about it!

Right, but the thing is that I want eventful to know about it so that I can query the event stream with getEvents using a time-based QueryRange. I don't see how this is possible without extending StoredEvent and QueryRange or creating new ones entirely. Unless I'm missing something?

You could also add timestamps to events explicitly and deal with generating them yourself. Just wrap your event type in a data type with UTCTime attached.

Problem with that is that filtering would then have to be implemented in the application rather than the database query, and will get quite heavy with a long event stream.

[jdreaver]

I don't see how this is possible without extending StoredEvent and QueryRange or creating new ones entirely. Unless I'm missing something?

My point is you don't need eventful to do this. If you are using a SQL store then eventful runs in the same Monad any persistent query could, so you can intermix eventful queries with other queries. Nothing is stopping you from adding a created_at column and doing SELECT * FROM events WHERE created_at > X; with persistent/esqueleto.

You can think of eventful as providing helper queries for you to use alongside other queries. eventful is not a gatekeeper to the event store that you must pass through.

Problem with that is that filtering would then have to be implemented in the application rather than the database query, and will get quite heavy with a long event stream.

That's not true. You can still use the database like you normally would. eventful is just creating a normal table. You define the columns and tell eventful about the columns it needs to know about.

• All eventful needs is a SqlEventStoreConfig

  eventful/eventful-sql-common/src/Eventful/Store/Sql/Operations.hs

  Lines 30 to 45 in 3f0c604

  	data SqlEventStoreConfig entity serialized =
  	SqlEventStoreConfig
  	{ sqlEventStoreConfigSequenceMakeEntity :: UUID -> EventVersion -> serialized -> entity
  	-- Key manipulation
  	, sqlEventStoreConfigMakeKey :: SequenceNumber -> Key entity
  	, sqlEventStoreConfigUnKey :: Key entity -> SequenceNumber
  	-- Record functions
  	, sqlEventStoreConfigUUID :: entity -> UUID
  	, sqlEventStoreConfigVersion :: entity -> EventVersion
  	, sqlEventStoreConfigData :: entity -> serialized
  	-- EntityFields
  	, sqlEventStoreConfigSequenceNumberField :: EntityField entity (Key entity)
  	, sqlEventStoreConfigUUIDField :: EntityField entity UUID
  	, sqlEventStoreConfigVersionField :: EntityField entity EventVersion
  	, sqlEventStoreConfigDataField :: EntityField entity serialized
  	}

• This is what the default Entity looks like:

  eventful/eventful-sql-common/src/Eventful/Store/Sql/DefaultEntity.hs

  Lines 25 to 47 in 3f0c604

  	share [mkPersist sqlSettings, mkMigrate "migrateSqlEvent"] [persistLowerCase|
  	SqlEvent sql=events
  	Id SequenceNumber sql=sequence_number
  	uuid UUID
  	version EventVersion
  	event JSONString
  	UniqueUuidVersion uuid version
  	deriving Show
  	|]
  	defaultSqlEventStoreConfig :: SqlEventStoreConfig SqlEvent JSONString
  	defaultSqlEventStoreConfig =
  	SqlEventStoreConfig
  	SqlEvent
  	SqlEventKey
  	(\(SqlEventKey seqNum) -> seqNum)
  	sqlEventUuid
  	sqlEventVersion
  	sqlEventEvent
  	SqlEventId
  	SqlEventUuid
  	SqlEventVersion
  	SqlEventEvent

---

There are more practical issues with timestamps to consider as well:

• What if someone doesn't want timestamps? How do we make them optional without having to pass a type parameter to every function/type in eventful?
• What timestamp representation do we use, both in the database and in eventful? Surely people will have different requirements.
• How do we generate timestamps for the in-memory stores? Does having timestamps mean every store/test must run in IO?
• How do we generate timestamps for DynamoDB? DynamoDB doesn't have a convenient way of generating timestamps server side, so we would have to trust all eventful clients to have the same system clock to be accurate.

---

Certainly timestamps on every event are useful as metadata. However, if you need timestamps as part of your business logic, they really should be included in the actual events. Then, if you want to do arbitrary time-based queries, you should have a read model that listens to all events from the main event store, and then projects them in such a way that time-based querying is easy and quick.