Honing the blade: Lessons from a year of event sourced systems design and implementation.
To the best of our knowledge, https://www.meetup.com/DDD-CQRS-ES/events/236265676/ is the only regularly occuring meetup in BC where participants can practice designing and buiding software. We focus on Domain Driven Design, CQRS and event sourcing because the craft of software development is mostly about managing coupling and cohesion and getting transactional boundaries right. There are other ways of achieving this, but DDD/CQRS/ES is usually the best choice. In our experience it gets the job done a lot more simply than many alternatives.
The meetup has run for most of 2016. During that year it evolved into doing series of three consecutive Thursday evenings, with a partial implementation of a new “real world” system during each series.
Usually it went like this:
Week 1: Event storming. A business friendly session where we determine what business objective the system should meet and what our scope is. The outcome is a complete spec in the form of series of events, commands and an initial idea what the aggregates are. (… a.k.a. state machines/transactional boundaries/actors/”microservices”. Maybe even objects. “Aggregates” is the Domain Driven design term. There are others in other contexts.).
Week 2: Implementation. We start building the system, one event at a time.
Week 3: Project management, infrastructure & operations. How do you get an event sourced system ready for production? How do you fix, evolve, scale, monitor? How do you leverage such an architecture in order to simplify project management and speed up delivery?
Some of the highlights:
– Hotel reservations. We were fortunate in having a participant who actually worked in a hotel. This is how we learned a hospitality industry dirty secret: That room is fresh and clean because they only clean it the day before you check in. No matter how long ago the last guest checked out … https://github.com/Adaptech/eventsourcing-hotelreservations
– SR & ED tax credit applications ( https://github.com/Adaptech/sreder ) . “Sreder” makes putting together claims easier. One person present in week 1 turned out to be very knowledgeable – this is how the event storming led to what might be a real world value proposition: We learned that often the fact that work was proving or disproving a hypothesis is only established with hindsight. What would distinguish “Sreder” from run-of-the-mill time tracking systems is to very easily allocate tracked time to new projects retroactively. Startup, anyone?
– Plumbing business resource optimization ( https://github.com/Adaptech/fullmoonplumbing ). We first thought the business owner was looking for a scheduling system, but event storming soon uncovered the core domain: The system turned out to be about answering the question “How do I avoid paying for unnecessary overtime?” Among other things, doing so required a scheduling capability. We ran this one longer than the usual three weeks, giving us time to dive into some more advanced modeling questions and to look at integrations. For the latter, we included a fictitious legacy system, simulated by a Google sheet.
What we learned
When we started it was all about the plumbing, the mechanics of building event sourced systems: How does CQRS and event sourcing work together? (Like this – https://adaptechsolutions.net/cqrs-es-or-is-it/ ). Do you get into publish-subscribe patterns right away? (yes) Do you do everything asynchronously with eventual consistency from the get-go? (No – keep it as simple as possible.) How do you deal with integrating legacy systems? (For example: https://adaptechsolutions.net/using-events-to-leverage-legacy-system-investments/ ).
It turned out that none of this mattered a lot.
Many of the newbie discussions in the field centre around these topics and assorted tooling (or lack thereof) and frameworks.
This isn’t what makes event sourced projects succeed or fail.
We came full circle – unsurprisingly, success or failure depend on the same two factors as in any other software implementation:
– Managing coupling & cohesion properly. Like in any other piece of complex machinery, getting the breakdown into subsystems right determines cost of ownership, timelines and complexity of enhancements and maintenance.
– Getting transactional scope right. Closely related to point 1. Much easier to to in CQRS systems with separate write- and read side. Basically, an event stream represents a transactional scope. Example: An order with a fictitious purchase limit: “For this order, you can add items up to a total of $1500. To avoid a 2nd user adding order items which bump the order amount beyond $1500, have “order” and “order item” in the same event stream, allowing (e.g.) optimistic locking. It’s obvious in this example, but oftentimes people feel that they need “sagas” or “controllers” to roll back transactions across multiple event streams. This can be a design smell, just like in any non-eventsourced system: Eventually, most of the system’s business logic can wind up in the “controllers”, relegating “business objects” to mere data access logic. In non-trivial systems under load, this “anaemic domain model” anti-pattern eventually forces a choice between locking everything down (only one operation at the time because the “controller” transactional scope isn’t as granular as it could be) and “improving performance” by abandoning transactions and hoping for the best, dealing with inconsistent data later. This antipattern happens whether you work event sourced or not. The only difference is how the fallout is detected and dealt with:
|Detect side effects||– Monitor / examine event logs.
– End users run into them.
|– End users run into them.|
|Recover from data inconsistencies||Compensating actions and/or fork & re-write event stream.
Re-generate read models
|DBA runs scripts to patch up the DB.|
It comes down to the same thing, event sourced or not: Unless we understand the basics of our craft, no amount of tooling and no design approach will prevent us building under-engineered hard to maintain systems, usually at a much slower pace.
– What matters is getting the domain model right.
– And to keep the plumbing as brainless and simple as possible. Because if it isn’t, people will prematurely optimize it instead of spending time in getting the domain model right.
In recognition of this we’ll spend 2017 focusing on more advanced scenarios in the meetup. In order to do so we’ll change the format:
– Six or more sessions per example system. This will allow us to model more complex real-world domains and experiment with versioning, fixes, unexpected requirements and various ways of doing infrastructure.
– Two different start times: Turn up at 5pm for the weekly 10,000ft level intro to event sourced systems and an overview of the current example. To dive right in, regulars or more experienced practitioners arrive at 5:30pm. This was a suggestion from our pre-holiday season retrospective, for addressing the need to cater to relative beginners as well as to our more frequent attendees. (Yes – some of you turn up at almost every session and we had some fantastic hires from that. Thank you for joining Adaptech – the enthusiasm and the skill is greatly appreciated!)
So that’s what we did in 2016. Join us at https://www.meetup.com/DDD-CQRS-ES/events/236265676/ . We look forward to another eventful year.
We are currently serving mulled wine because of the un-British-Columbia like cold weather.
About The Author & Adaptech:
Robert Reppel is Director of Engineering at Adaptech. Adaptech Solutions partners with its clients to ensure the success of their DDD/CQRS/ES projects. Sometimes it’s as simple as introducing an organization to the concepts all the way through to architectural guidance, domain-modelling and building the software. Our goals are to reduce the risk of costly and time-consuming re-writes, ensure that projects are successful and that the client team looks brilliant. Adaptech is based in Vancouver, Canada, has an office in Seattle, Washington and does business around the world.