Posts in "Architecture"

Architecture Clojure DDD

DDD Layered architecture in Clojure: A first try

4 Comments

The first step in my effort to freshen up our time tracker using DDD & Clojure has been finding a way to structure my code. Since I don’t have that much Clojure experience yet, I decided to take the DDD layered architecture and port it as directly as possible. This probably isn’t really idiomatic Clojure, but it gives me a familiar start. This post should be regarded as such, my first try. If you know better ways, don’t hesitate to let me know.

The architecture

As a picture is worth a thousend words:
layered
This architecture is mostly the same as the one advocated in the DDD Blue Book, except that the Domain Layer does not depend on any data-related infrastructure and there’s a little CQRS mixed in. I think this is mostly standard these days. In this design, the application layer is responsible for transaction management. The ‘Setup’ part of the UI layer means setting up things like dependency injection.

In this post I’ll focus on the interaction between application services, domain objects, repositories and the data layer. I’ll blog about other parts (such as validation) in later posts.

The domain

Unfortunately, I’m not able to release the code for the time tracker just yet (due to some issues with the legacy code). So for this post I’ll use an example domain with curently just one entity… Cargo 🙂 The Cargo currently has one operation: being booked onto a Voyage.

The approach

Let’s start with the Domain Layer. Here, we need to define an “object” and an “interface”: the Cargo and CargoRepository respectively.

Cargo entity

The Cargo entity is implemented as a simple record containing the fields cargo-id, size and voyage-id. I’ve defined a constructor create-new-voyage which does its input validations use pre-conditions.

There’s one domain operation, book-onto-voyage which books the cargo on a voyage. For now, the requirement is that it can’t already be booked on another Voyage. (Remember this post is about overall architecture, not the domain logic itself, which is for a next post).

Furthermore, there is a method for setting the the cargo-id since we rely on the data store to generate it for us, which means we don’t have it yet when creating a new cargo.

Here’s the code:

Cargo Repository

The Cargo Repository consists of 2 parts: the interface which lives in the domain layer, and the implementation which lives in the data layer. The interface is very simple and implemented using a Clojure protocol. It has 3 functions, -find, -add! and -update!.

A note about concurrency: -find returns both the cargo entity and the version as it exists in the database in a map: {:version a-version :cargo the-cargo}. When doing an -update! you need to pass in the version so you can do your optimistic concurrency check. (I’m thinking of returning a vector [version cargo] instead of a map because destructuring the map every time hurts readability in client code, I think.)

Furthermore, I’ve defined convenience methods find, add! and update!, which are globally reachable and rely on a call to set-implementation! when setting up the application. This is to avoid needing to pass (read: dependency inject) the correct repository implementation along the stack. This is probably a bit controversial (global state, pure functions, etc), and I look forward to exploring and hearing about alternatives.

Cargo repository MySQL implementation

I’m using MySQL as the data store, and clojure.java.jdbc for interaction with it. The cargoes are mapped to one table, surprisingly called cargoes. I don’t think there’s anything particular to the implementation, so here it goes:

The final parts are the Application Services and the UI.

The Application Service

I never have good naming conventions (or, almost equivalently, partitioning criteria) for application services. So I’ve just put it in a namespace called application-service, containing functions for all domain operations. The operations can be taken directly from the Cargo entity: creating a new one, and booking it onto a voyage. I use the apply construct to invoke the entity functions to avoid repeating all parameters.

Code:

The UI The tests

To not make this post any longer than it already is I’m not going to show a full UI, but a couple of tests exercising the Application Service instead. This won’t show how to do queries for screens, but for now just assume that I more or less directly query the database for those.

There isn’t much to tell about the tests. If you are not that familiar with Clojure, look for the lines starting with deftest, they define the actual tests. The tests show how to use the application service API to handle commands. They test the end result of the commands by fetching the cargo from the repository and checking its state. I use the MySQL implementation for the database, since I already have it and it performs fine (for now).

Conclusion

The code in this post is pretty much a one-to-one mapping from an OO kind of language to Clojure, which is probably not ideal. Yet, I haven’t been able to find some good resources on how you would structure a business application in a more Clojure idiomatic way, so this will have to do. Nevertheless, I still like the structure I have now. I think it’s pretty clean and I don’t see any big problems (yet). I look forward to exploring more alternatives in the next couple of months, and I’ll keep you updated.

All code (including tests) is available on GitHub.

Architecture Logical vs Phyiscal

The Logical vs the Physical: Layered Architecture

0 Comment

A lot of mistakes and confusion in software development arise from conflating logical and physical concerns. For example, an object in a conceptual (logical) object model would usually be implemented by multiple objects in the implementation (physical) model. Not knowing this distinction might entice you to include all the implementation classes into the conceptual model, or not factor your implementation classes neatly. One layer down, this mistake is made again when mapping your objects to a data model.

This kind of mistake is so common, that I decided to try to write a little bit about it whenever I see this happening in practice. In this first post about it, I’ll talk about it in the context of Layered Architecture.

Layered architecture

I don’t think the layered architecture style (or its cousin, hexagonal architecture) needs much of an introduction:
layered
The UI layer is responsible for visualizing stuff, the application layer for managing the application state, the domain layer for enforcing business rules and the data layer manages the data. Every layer builds on top of the layers below it. It can know about those lower layers, but not about the layers above it. Depending on your point of view the data layer is sometimes thought of as an infrastructure layer which runs besides all the other layers. For the purpose of this discussion, that doesn’t matter much.

Logical vs Physical

So is the separation of layers a logical or a physical separation? This question raises another question: what it means to belong in a certain layer. Am I in the application layer because my code is in a namespace called MyAwesomeApp.AppLayer, or am I in the application layer because I behave according to its rules: I don’t know about the UI layer, and I don’t enforce business rules or do data management.

When it’s stated like that, you’re probably going to agree that it should be the latter. It doesn’t matter that much where you’re physically located, but only what you logically do. Yet, this is completely contrary to what I encounter in a lot of code bases.

A common sign is having your projects (or JARS, or packages, or namespaces, or whatever) named after their layers: MyAwesomeApp.UI, MyAwesomeApp.Application, MyAwesomeApp.Domain, MyAwesomeApp.DAL, etc. The UI package would contain your HTML, JavaScript and web endpoint code. The application layer would contain a couple of application services. The Domain layer would host our business rule code, presumably through a domain model. Finally, the DAL would do all the interactions with a given database.

So when is this a problem? Well, it doesn’t really have to be a problem, as long as you don’t need to do any, say, business logic in your UI or DAL package. But how often is this the case? In almost every project I work in there’s at least a bit of client-side (JavaScript) validation going on, which is definitely a business rule. Yet, this code lives in the UI layer. Or, I might have a unique username constraint in my database (which is part of the DAL), which is also a business rule. I’m pretty sure this happens in any project of practical size.

Now, this isn’t really a big problem in itself either. There’s the problem of the naming being off since there’s business logic in the UI and DAL package, which you would expect in the domain layer and this probably causes some confusion to new-comers. Secondly, we might ‘forget’ applying a layered design inside the individual layers since it looks like we’ve given up on this architecture for this feature anyway (since we’re not putting it in the appropriate package). That causes that part of the code to be less well-designed.

A real problem occurs, however, when we dogmatically try to put the business rules in the domain layer anyway, an idea often strengthened by the (dogmatic) need to keep things DRY. Ways I’ve seen this gone and done wrong are at least the following:

  • Putting JavaScript code in our domain package, and somehow merging that to the rest of JS code at build time. The causes major infra-headaches.
  • Enforcing unique constraints and the like in memory. This doesn’t perform.
  • Refrain from using purely client-side validations, and always go through the backend via ajax. Users won’t like the latency.
  • Doing a lot of pub/sub with very generic types to be able to talk to another layer you shouldn’t know about anyway. In this code base, I have no idea what’s going on at runtime anymore.
  • Endless discussions among developers to agree on where a given piece of code should go. This is just a waste of time.

And these are probably just the tip of the iceberg.

We can prevent this problem by allowing our packages to not map 1-to-1 to the physical layers. In this case it’s an easy discussion where code should go. Now, we need to apply a little more discipline within this package to still apply a layered architecture, but at least it will be an honest story. Another method is to package based on the role a given set of code has: one package has to do with client facing concerns, another with database concerns and yet another with a domain model. You might think these correspond very closely to UI/APP/Domain/Data layer, and you’d be right, but the intention is different. The fact that one package handles database concerns also means it can be responsible for handle business rules that can most efficiently be implemented in a database.

This way, your physical package structure doesn’t have to reflect the logical layers of your layered architecture. It’s OK to have some business rules in your JavaScript or database, and they still belong to the domain layer when you do. However, make sure you still maintain the constraints of the layered architecture within that physical package as well.

Conclusion

This is just one example where conflating logical and physical concerns can cause big problems in your code base. By untangling both, we can feel less dirty about ourselves when we put a certain piece of code in the ‘wrong’ layer and at the same time build more effective software.

.NET Architecture

.NET is slowing me down

0 Comment

.NET has been my primary development environment for a little over 5 years now. I’ve always really liked it, had a lot of success with it, and learned a lot while using it. The tooling and maturity of the platform is, and has been, right where it had to be for me. In a lot of projects, it allowed me to really focus on the domain and I seldom have to write custom tooling for doing standard stuff, which I had to on other platforms. This allowed me to deliver a lot of value to my clients, and they’re happy about that.

There is, however, a problem that’s growing bigger and bigger with .NET: it’s getting slow to develop on. This doesn’t mean .NET itself is getting slow, it means the developer experience is getting slower. To illustrate my point, I’ve measured the times to the very first byte (that is, the time after a rebuild) for the template application for various MVC versions:

.NET Version MVC Version First available date Time to very first byte (seconds)
4.0 2 March 2010 1.00
4.0 3 January 2011 1.12
4.5.2 3 May 2014 1.45
4.0 4 August 2012 2.63
4.5.2 4 May 2014 2.89
4.5.2 5.2.3 January 2015 3.47
4.6 5.2.3 July 2015 3.58
4.6 6.0.0-beta5 July 2015 1.89

So, over the course of 5 years the time to load the first page has increased by a factor 3.5, and 2.5 seconds in absolute terms. Now, it seems ASP.NET 5 is going to reduce times a bit, but still not to the 2010 level.

To make matters worse: something like Entity Framework is getting equally slower, and hitting a page that goes to the database might easily take somewhere between 5-10 seconds. The same goes for tests: running the first easily takes a couple of seconds due to EF only.

Environmental Viscosity

So, what’s the problem? Environmental viscosity. To quote Uncle Bob from PPP:

Viscosity of the environment comes about when the development environment is slow and inefficient. For example, if compile times are very long, developers will be tempted to make changes that don’t force large recompiles, even though those changes don’t preserve the design.

This is exactly what’s going on here. Because load times are slow, I tend to:

  • Make bigger changes before reloading
  • Write less tests
  • Write tests that test larger portions of functionality
  • Implement back-end code in the front-end (HTML/JavaScript)
  • Visit reddit while the page loads

All these things are undesirable. They slow me down, and compromise the quality of the software. If you ever worked with “enterprise” CMS software, you’ve seen this happen to the extreme (I sure have): there might be minutes between writing a change and the page actually being loaded.

Even if you don’t do all the above the things, and slavishly wait for the page to load/test to run every time, you’re still wasting your time, which isn’t good. You might not recognize it as being a big deal, but imagine making 500 changes every day, that translates to 500 x 5s = 2500 seconds of waiting. That’s more than 40 minutes of waiting, every day.

Architecture

To reiterate: slow feedback compromises software quality. What I want, therefore, is feedback on my changes under a second, preferably under 500ms. My choice of technology/tools will definitely factor in this requirement, and it will be a strong factor.

For example, my choice for data access defaults to Dapper these days, because it’s just much faster than EF (tbf, I also rely less on “advanced” mappings). Even something like PHP, for all its faults, tends to have a time-to-very-first-byte that’s an order of magnitude faster than .NET apps, and therefore be something I might consider when other .NET qualities aren’t that important.

To me, the development experience is as much part of software architecture as anything else: I consider anything related to building software a part of architecture and since slow feedback compromises software quality, the development experience is certainly part of architecture.

The future of .NET

I certainly hope Microsoft is going to improve on these matters. There is some hope: ASP.NET 5 and Entity Framework 7 are right around the corner, and they promise to be lighter-weight, which I hope translates in faster start-up times. Also, Visual Studio 2015 seems to be a bit faster than 2013 (which was and is terrible), but not as fast a VS2012. I guess we’ll have to wait and see. For the time being, though, I’ll keep weighing my options.

Architecture REST

Integrating application data using Atom feeds

2 Comments

While Atom is often associated with things like blogs or news feeds, it turns out it’s also an excellent vehicle for integrating application data across your systems. At Infi we’re using this technique for several integration scenario’s for our customers, and it’s quickly becoming a preferred way of solving this kind of problem.

This post assumes a basic understanding of Atom (though you can probably follow it if you don’t). If you need to brush up your knowledge, you can read a basic introduction here.

Context

We often have a situation where a subset of users manage data that is exposed to a much larger set of different users. For example, we might have a website that deals with selling used cars from car dealers, where the car dealers manage the car data, such as pictures, descriptions, etc. This data is then viewed by a large group of users on the website. One of our customers has a similar model, though in a different domain.

As this customer has grown, more and more additional systems needed access to this data, such as several websites, external API’s and an e-mail marketing system. We used to do these integrations on an ad-hoc basis, using a custom-built solution for every integration (e.g. database integration, RPC, XML feeds). As you can imagine, this situation was becoming an increasing point of pain since it doesn’t really scale well in a couple of dimensions, such as performance, documentation, reward/effort, etc. The situation looked more or less like this:

current-status

Requirements

To remove some of this pain we decided on building something new, specifically having the following requirements:

Good performance. This means 2 things: first, we want low latency responses on requests on our integration system. Second, we don’t want high loads on the integration solution pulling down other parts of our system.

Low latency updates. It’s OK for the data to be eventual consistent, but under normal conditions we want propagation times to be low, say on the order of several minutes or less.

No temporal coupling. Consumers shouldn’t be temporally coupled to the integration system. This means they should be able to continue doing their work even when the integration endpoint is unavailable.

Support multiple logical consumers. All (new) integrations should be done through the new system, such that we don’t need to do any modifications for specific consumers.

Ease of integration. It should be easy for consumers to integrate with this new system. Ideally, we should only have to write and maintain documentation on the domain-specific parts and nothing else.

It turns out we can address all these requirements by combining two ideas: snapshotting the data on mutations and consumer-driven pub/sub with Atom feeds.

Solution

The core idea is that whenever an update on a piece of data occurs, we snapshot it, and then post it to an append-only Atom feed:

handling_mutation

This way we essentially build up a history of all the mutations inside the feed. The second part of the solution comes from organizing (paging) the feed in a specific way (this builds on top of the Feed Paging and Archiving RFC):

structure

In this case we divide the snapshots into pages of 20 entries each.

The value in this structure is that only the root and the currently ‘active’ pages (i.e. not completely filled pages) have dynamic content. All the other resources/URLs are completely static, making them indefinitely cacheable. In general, most of the pages will be full, so almost everything will be cacheable. This means we can easily scale out using nginx or any other caching HTTP proxy.

Consuming the feed

From a consumer point of view you’re gonna maintain your own copy of the data and use that to serve your needs. You keep this data up to date by chasing the ‘head’ of the feed. This is done by continuously following the rel=”previous” link in the feed documents.  Once you read the head of the feed (indicated by no entries, and a missing link rel=”previous”) you keep polling on that URL until a new entry appears.

Evaluating the solution

To see how this solution fulfills our requirements, let’s revisit them:

Good performance 

  • Since data is snapshotted, we don’t need to do expensive querying on our transaction-system database. This allows for both quick responses to in the integration system as well as isolation of our transactional system.
  • Because almost everything is cacheable, you can easily scale out and provide low-latency responses.

Low latency updates

  • The detection of changes to the data is event-based instead of some sort of large batch process to detect changes. This allows the data to flow quickly through the system and appear in the feed. Polling the head of the feed is not expensive and can therefore be done on minute or even second basis, so the clients themselves will also be able to notice the updates quickly.

No temporal coupling

  • First of all, the consumers are decoupled from the integration system because they have their own copy of the data so they don’t have to query the feed in real-time to serve their needs. Secondly, the integration system itself is also not coupled to the system containing the original data, since the snapshots are stored in the Atom feed.

Support multiple logical consumers

  • Multiple logical consumers are trivially supported by handing out the URL to the feed endpoint.
  • One problem is different consumers requiring different parts of the data.  Currently, we’ve solved this by always serving the union of all required data pieces to all the consumers. This isn’t exactly elegant, but it works (though we sometimes need to add fields). A better solution would be for clients to send an Accept header containing a mediatype that identifies the specific representation of the data they want.
  • We also built in rudimentary authentication based on pre-shared keys and HTTP basic authentication.

Ease of integration

  • This is where the whole Atom thing comes in. Since both Atom and paging techniques are standard, we only need to document the structure of our own data. From a client point of view, they can use any standard Atom reader to process the data.
  • To make things even more easy to integrate, we also created a custom JSON representation for Atom. This is useful for consumers on platforms that don’t have strong XML support.

Conclusion

As you can see, all our requirements are met by this solution. In practice, it also works very well. We’ve been able to reduce loads on our systems, get data quicker into other systems, and both for us and our partners it’s easier to implement integrations. We started out doing this for just one type of data (property information), but quickly implemented it for other types of data as well.

One of the challenges is pick the right level of granularity for the feeds.  Pick the granularity too narrow, and it becomes harder for a client to consume (needs to keep track of lots of feeds). Pick it too wide, and there will be a lot of updates containing little change. In our cases, common sense and some basic analysis of how many updates were expected worked out fine.

The main drawbacks we encounter are twofold. First, people not familiar with Atom and the paging standards sometimes had problems working with the feed structure. Especially when there’s no platform support for Atom or even basic XML, we sometimes still have to help out. Second, for some integrations maintaining a copy of the data on the consumer side proved to be a bit too much (or not even possible). For these situations we actually built a consumer of our own, which served data to these thinner clients.

Credits

The ideas in this post are not new. We were particularly inspired by EventStore, which also provides an Atom based API for exposing its event streams.

Architecture

When not to do responsive design

2 Comments

The premise of responsive webdesign is great: by using one set of HTML, CSS and JavaScript for all device types you can shorten development time and costs compared to having multiple. While I haven’t seen any proof of this, I can imagine this being so in certain situations. In general, however, I don’t think this is the case and there are a lot of situations where responsive can actually increase your development time and costs.

As part of my job, I regularly get asked by clients whether they should go responsive. My usual answer is that they probably shouldn’t, for various reasons. In this post I’ll list some of them.

You want to test new features often

As a first case let’s say you’re introducing a new feature, and are not yet completely sure if your users are gonna value it. In this case it doesn’t make sense to directly go all the way. Instead, you want to put something into the market as quickly as possible, and when it turns out your feature is actually a hit, you want to implement it for other platforms.

Clients are often surprised by this reasoning, since it’s assumed that the whole point of responsive is that you have to do the work only once. Well, that’s just not true. No matter how you do it, you will need to design how the scaling up/down will actually work; how the pages are laid out at the different sizes, how the interaction works. This will always be extra work, apart from any coding activities. On the coding side, there will still be extra development work since building HTML/CSS that works for a variety of devices types will always be harder than building it for a single device type. Finally, you will need to test it on all the device types.

So going responsive increased your time to market, but what you wanted is to test the feature ASAP, instead of on all your users.

You want to support a new class of device

You already have a mobile site, but now want to add a desktop version. You have two options here: build separate desktop HTML/CSS or rebuild both the desktop and mobile site as a responsive site.

I’d always go for the first option because of the following implications when (re)building both together:

  • I get more development work since I now need to implement two design instead of one.
  • The mobile site needs to be retested and debugged.
  • I can probably not release both independently. That means I probably need to wait with making changes to the mobile site until I’ve released the desktop site, which might take a while.

In short, it’s gonna take more time to do a desktop version and you’re holding back mobile development, while I’m not convinced there is any benefit. Adding just the desktop version also goes hand in hand with the XP principle of baby steps, changing one thing at a time, which I strongly believe in.

One thing I might do is take into consideration that when I’ve finished the desktop version, I’m gonna retrofit that as a responsive version of the mobile site. But only when I can see the benefits of going responsive (see conclusion).

The use cases vary greatly among device classes

When I’m buying something online, I usually do this on a desktop. I’d like to browse a little bit, compare, read some reviews, etc, before making the purchase. After making the purchase, I then use my phone to regularly check up on the status of my order.

It’s pretty obvious these are two completely different use cases, and therefore require completely different interaction models and navigation paths. This is gonna be hard to do when you’re doing responsive, since responsive assumes pretty much a 1-to-1 mapping to what you can do on each page, and how you navigate. So again, without a clear benefit, you’ve seriously constrained your own ability to provide the most appropriate user experience for each device type.

You want separate teams handling mobile and desktop

As stated above, since use cases probably vary among device classes, I might want to have separate teams handling the development of each, both of them optimizing for their specific type of use cases. I want those teams to be autonomous. Having them work in the same code base is not gonna make that work: there needs to be a lot of coordination to avoid breaking each other’s work, and you can never release independently. So using responsive hurts your ability to scale/structure your teams as you like.

Conclusion

To be fair, most of the problems listed above can actually be circumvented if you try hard enough. Doing that, however, nullifies the entire argument for doing responsive in the first place, which is saving time/costs.

The underlying problem with all of the above cases is that you’re introducing coupling, and we all know that the wrong kind of coupling can really hurt your code. In the above examples, the coupling is working against you instead of for you, manifesting itself in less flexibility, less agility, longer times to market and a lesser end user experience. All this without any real, clear benefit. For me, that’s hard to justify. Especially since that, in my experience, it’s not that hard to build either a separate desktop or mobile variant of your site once you already have the other. Most of the time actually goes into other work, such as settling on functionality, implementing that functionality, designing basic styles (which you can share), etc. I think in a lot of situations this will actually save development time/costs compared to going responsive.

Only in situations where you have a very basic design and just a small amount of functionality that’s not going to change a lot, and you don’t care about flexibility a lot, responsive might actually reduce development time, but certainly not a lot (I dare say at most 5%, if you do reuse basic style components).

Don’t get me wrong, I strongly feel you should provide a good experience for as big an audience as possible. I just don’t think responsive (across all device classes) is the general way to do it.

On URLs

Some people think the only way to get the same URL for desktop and mobile is doing responsive. This is not true since you can detect device class server-side and then decide which HTML you’re gonna serve. And really, Google doesn’t care.

Afterthought: How this relates to MVC

MVC has emerged as method to have multiple views on the same data. Jeff Atwood once wrote an article that on the web, HTML can be seen as the Model and CSS as the View. I don’t agree. For me, HTML is part of the view. To show multiple representations of the same data (the model), as you do when viewing on multiple devices, you create multiple views, comprising both HTML and CSS.