Z2-environment Version 2.5 Is Out

It took a while, but it got finally done. Version 2.5 of the z2-environment is out. Documentation and samples have been updated and tested.

Here is what version 2.5 was about:

More Flexibility in Component Configuration

A major feature of z2 is to run a system cluster strictly defined by centrally, version-controlled configuration. As there is no rule without an exception, some configuration is just better defined by the less static and local system runtime environment, such as environment variable or scripted system properties.

To support that better and without programming, component properties may now be expressed by an extensible expression evaluation scheme with built-in JEXL support. Expressions are evaluated upon first load of component descriptors after start or after invalidation at runtime.

Some use-cases are:

  • Seamless branch or URL switching based on environment settings.
  • Dynamic evaluation of database config, in particular remote authentication based on custom evaluators or environment settings.
  • Declarative sharing of configuration across components.

Some aspects, such dynamic evaluation of searchable properties, were not made dynamic due to the risk of unpredictable behavior. Future work may show that the concept can be extended further though.


Check it out in the documentation.

More Complete in-Container Test Support

Z2 offers a sleek, built-in way of running application-side in-container tests: z2Unit. Previously, the JUnit API had its limits in serializabilty over the wire – which is essential for z2Unit. JUnit improved greatly in that department and after the correspinding adaptation of z2Unit some problematic Unit Test Runner combinations (such as e.g. z2Unit and parameterized tests) now work smoothly.


Check it out in the documentation.

Better Windows Support

Some very old problems with blanks in path or parameter names got finally fixed. There is a straight forwared command line specification syntax for worker processes that is (mostly) backward compatible.

Also, and possibly more importantly, system property propagation from Home to Worker processes is now fully configurable.

Check it out in the documentation.

Better Git Support

Z2 can read directly from GIT repositories. However, previously only a branch could be specified as content selector. Now any GIT ref will do.


Check it out in the documentation.

There is some more. Please check out the version page, if you care.

What’s next:

The plans for 2.6 are still somewhat open. As the work in Version 3 will not make it into any production version – namespace changes are too harming at this time – some useful structural simplifications implemented in 3.0 are considered, such as:

  • Worker processes as participants in Home process target states (rather than a Home Layout)
  • Introducing a “root” repository that hosts any number of remote or “inline” repositories and so streamlines local core config
  • Supporting a co-located Tomcat Web Container as an alternative to an integrated Jetty Web Container
  • Component templates that provide declarative default configurations and so remove duplications (i.e. any Java component based on Spring+Spring AOP).

Thanks and good luck an whatever you do that needs to be done right!




Java Modularity – Failing once more?

Like so many others, I have pretty much ignored project Jigsaw for some time now – assuming it would stay irrelevant for my work or slowly fade away and be gone for good. The repeated shifts in planned inclusion with the JDK seemed to confirm this course. Jigsaw started in 2009 – more than six years ago.

Jigsaw is about establishing a Java Module System deeply integrated with the Java language and core Java runtime specifications. Check out its goals on the project home page. It is important to note the fourth goal:


Make it easier for developers to construct and maintain libraries and large applications, for both the Java SE and EE Platforms.


Something Missing?

Lately I have run into this mail thread: http://permalink.gmane.org/gmane.comp.java.openjdk.jigsaw/2492

In that mail thread Jürgen Höller (of Spring fame) notes that in order to map Spring’s module layout to Jigsaw modules, it would be required to support optional dependencies – dependencies that may or may not be satisfied given the presence of another module at runtime.

This is how Spring supports its set of adapter and support types for numerous other frameworks that you may or may not use in your application: Making use of Java’s late linking approach, it is possible to expose types that may not be usable without the presence of some dependent type but will not create a problem unless you start using them either. That is, optional dependencies would allow Spring to preserve its way of encapsulating the subject of „Spring support for many other libraries“ into one single module (or actually a jar file).

In case you do not understand the technical problem, it is sufficient to note that anybody who has been anywhere near Java class loading considerations as well as actual Java application construction in real live should know that Spring’s approach is absolutely common for Java infrastructure frameworks.

Do Jigsaw developers actually know or care about Java applications?

Who knows, maybe they simply forgot to fix their goals. I doubt it.


Module != JAR file


There is a deeper problem: The overloaded use of the term module and the believe of infrastructure developers in the magic of the term.

Considering use of the module term in programming languages, it typically denotes some encapsulation of code with some interface and rules on how to expose or require some other module. This is what Jigsaw focussed on and it is what OSGi focussed on. It is what somebody interested in programming language design would most likely do.

In Java this approach naturally leads using or extending the class loading mechanism to expose or hide types between modules for re-use (or information hiding resp.) which in turn means to invent descriptors that describe use relationships (meaning the ability to reference types in this case) and so on.

This is what Jigsaw does and this is what OSGi did for that matter.

It is not what application developers care about – most of the time.

There is an overlap in interest of course. Code modules are an important ingredient in application assembly. Problems of duplicate type definitions by the same name (think different library versions) and separation of API and implementation are essential to scalable, modular system design.

But knowing how to build a great wall is not the same as knowing how to build a great house.

From an application development perspective, a module is much rather a generic complexity management construct. A module encapsulates a responsibility and in particular should absolutely not be limited to code and is not particularly well-served by squeezing everything into the JAR form factor.

What we see here is a case of Application vs. Infrastructure culture clash in action (see for example Local vs. Distributed Complexity).

The focus on trying to find a particularly smart and technically elegant solution for the low-level modularization problem eventually hurts the usefulness of the result for the broader application development community (*).

Similarly, ignorance of runtime modularization leads to unmaintainable, growth-limited, badly deployable code bases as I tried to describe in Modularization is more than cutting it into pieces.

The truth is somewhere in between – which is necessarily less easily described and less universal in nature.

I believe that z2 is one suitable approach for a wide class of server-side applications. Other usage scenarios might demand other approaches.

I believe that Jigsaw will not deliver anything useful for application developers.

I wish you a happy new year 2016!



* One way of telling that the approach will be useless for developers is when discussions conclude that “tools will fix the complexity”. What that comes down to is that you need a compiler (the tool) to make use of the feature, which in turn means you need another input language. So who is going to design that language and why would that be easier?

* It is interesting to check out the history of SpringSource’s dm Server (later passed on to R.I.P. at Eclipse as project Virgo). See in particular the interview with Rod Johnson.

Z2 as a Functional Application Server


As promised this is first in a series of posts elaborating on integration of Clojure with Z2. It probably looks like a strange mix, however I believe it’s extremely empowering combination of two technologies sharing lots of design philosophy. Clojure has brought me lots of joy by enabling me to achieve much more in my hobby projects and I see how the combination of z2 and Clojure further extends the horizon of what’s possible. I’d be happy if I manage to help other people give it a try and benefit in the same way I did.

The LISP universe is very different one. It’s hard to convince someone with zero previous experience to look at the strange thing with tons of parentheses written using polish notation, so I am going to share my personal story and hope it resonates with you. I will focus on the experience or how I “felt” about it. There is enough theory and intellectual knowledge on the internet already and I will link to it where appropriate.

So, given this is clearly personal and subjective view, let’s put in some context.

Short Bio

I’ve been using Java professionally for 12+ years. Predominantly in the backend. I’ve worked on application servers, as well as on business applications in large, medium and small organizations. Spring is also something I have been heavily relying on in the last 8 years. Same goes for maven. I’ve used JBoss and done bit of application server development myself, but when Spring Boot came up I fell in love with it.

Like every other engineer out there my major struggle through the years have been to manage the complexity. The inherent complexity of the business problem we have to solve plus the accidental complexity added by our tools, our poor understanding of the problem domain and our limited conceptual vocabulary. I have been crushed more than once under the weight of that complexity. Often my own and the team’s share would be more than 50%. I have seen firsthand how poorly groomed code base ends up in state where the next feature is just not possible. This has real business impact.

The most scary thing about complexity is that it grows exponentially with size. This is why I strongly subscribe to the “code is liability” worldview. Same goes for organizations. The slimmer you are the faster and further you can go.

Ways to deal with complexity

Now that the antagonist is clearly labeled, let’s focus on my survival kit.

#1 Modularization

One powerful way to get on top of complexity is divide and conquer by using modularization. This is where z2 comes into the game. It has other benefits as well, but I would put it’s modularization capabilities as feature #1. Maven and Spring have been doing that for me through the years. On more coarse level Tomcat and JBoss provide some modularization facilities as well, however it is extremely rare in my experience where they are deliberately exploited.

Getting modularization right is hard on both ends:

  • The framework has to strike a balance between exercising control and enabling extensibility, otherwise it becomes impractical.
  • The component developers still have to think hard and define boundaries of “things” while using the framework idioms with mastery. I haven’t met yet technology that removes this need. It’s all about methodology and concepts (I dislike the pattern cult).

Discussing more precise definition of what exactly is modularization and why the well-known methodologies are too general to be useless as recipes is too big of discussion for here.

My claim is that z2 strikes the best balance I have seen so far while employing very powerful concept.

#2 Abstractions

Another powerful way is to use better abstractions. While modularization puts structure in chaos, the right abstractions reduce the amount of code and other artifacts, hence the potential for chaos. Just like any other thing, all abstractions are not made equal and I assume they can be ordered according to their power.

My personal definition for power: if abstraction A allows you to achieve the same result with less code than abstraction B then it’s more powerful. Of course reality is much more hairy than this. You have to account for abstraction’s implementation, investment in learning, long term maintenance costs & so on.

Alternative definition: if abstraction A allows you to get further in terms of project size and complexity (before the project collapses) it’s more powerful.

The abstractions we use on daily basis are strongly influenced by the language. Language can encourage, discourage (due ergonomics) or even blacklist an abstraction by having no native support for it. My claim here is that the Java language designers have made some very limiting choices and this has profound effect on the overall productivity as well as the potential access to new abstractions. Clojure, on the other side has excellent mix right out of the box with convenient access to very wide range of other abstractions.

The OO vs. FP discussion deserves special attention and will get it. I won’t claim that Clojure is perfect, far from it. However the difference in power I have experienced is significant and big part of that difference is due to carefully picked set of base abstractions implemented in very pragmatic way.

 So, what’s next?

Next comes the story how Java and DDD helped me survive and how JavaScript made me feel like a fool for wasting so many hours slicing problems the wrong way and worrying about insignificant things. Clojure will show up as well, you can count on this.

While you wait for the next portion, here are two links that have influenced heavily my current thinking:

  • Beating the averages — the blub paradox has been an eye opening concept for me. I have read this article in 2008 for the first time and kept coming back to it. It validated my innate tendency to be constantly dissatisfied with how things are and look for something better. Paradoxically, it never made me try out LISP 🙂
  • Simple made easy — This is the presentation that among other side effects made me give Clojure a chance. This presentation probably has the best return of investment for an hour spent in front of the screen.

Continuity is King

towerUnfortunately there has been so much going on in my work life and my private life lately that I didn’t get around thinking and writing much.

Here is just a short note that v2.4 of z2 is ready: v2.4

It simply upgrades z2 to Java 8 and upgrades the version of Jetty we use to 9.3. The latter implies that Java 8 is a strict requirement too.

Little change is good as those projects that run with z2 need anything but platform disruption at this time.

As the core did not change incompatibly (at least not that I know), using a v2.4 core with previous z2-base version 2.3 will simply add Java 8 support to any such setup as well.

Anyway here’s what I hope to continue on once I am back to normal operations again:

  • A piece on how to use clojure on z2 with a guest author from Vienna
  • A piece on a variation of the Stockholm Syndrome that can be observed in the relationship of developers with their toolset
  • A piece on how organization can be classified as Project vs. Product driven


Microservices Nonsense

Microservice Architecture (MSA) is a software design approach in which applications are intentionally broken up into remoteable services, in order built from small and independently deployable application building blocks with the goal of reducing deployment operations and dependency management complexity.

(See also Fowler, Thoughtworks)

Back in control

Sounds good, right? Anybody developing applications of some size knows that increasing complexity leads to harder to manage updates, increased deployment and restart durations, more painful distribution of deployables. In particular library dependencies have the tendency to get out of control and version graphs tend to become unmanageable.

So, why not break things up into smaller pieces and gain back control?

This post is on why that is typically the wrong conclusion and why Microservice Architecture is a misleading idea.

Conjunction Fallacy

From a positive angle one might say that MSA is a harmless case of a conjunction fallacy: Because the clear cut, that sounds more specific as a solution approach, makes it more plausible (see the Linda Problem of ….).

If you cannot handle it here, why do you think you can handle it here?

If you cannot organize your design to manage complexity in-process however, why should things work out more smoothly, if you move to a distributed setup where aspects like security, transaction boundaries, interface compatibility, and modifiability are substantially harder to manage (see also Distributed big ball of… )

No question, there can be good reasons for distributed architectures: Organization, load distribution, legacy systems, different expertise and technology preferences.

It’s just the platform (and a little bit of discipline)

Do size of deployables and dependency management complexity belong into that list?

No. The former simply implies that your technology choice has a poor roll-out model. In particular Java EE implementations are notoriously bad at handling large code bases (unlike, you might have guessed, z2). Similarly, loss of control over dependencies shows a lack of dependency discipline and, more often, a brutal lack of modularization effort and capabilities (see also Modularization is….)

Use the right tool

Now these problems might lead to an MSA approach out of desperation. But one should at least be aware that this is platform short-coming and not a logical implication of functional complexity.

If you were asked to move a piece of furniture you would probably use your car. If you were asked to move ten pieces of furniture, you would not look for ten cars – you would get a truck.



Working on z2env Version 3

Despite its fantastic qualities as a development and execution environment, z2’s adoption is very low. That of course does not at all stop us from improving it further (as we are actively benefiting from it anyway).

Whenever I talk about z2, the feedback is typically in one of two categories.

The first one is the “I don’t get it”-category.

There was a time when running builds was such a natural ingredient of software development to me, that I would have been in that category as well. So I forgive them their ignorance.

The other category is the “Great idea – … to bad I cannot use it”-category.

Being a disruptive approach and knowing how change averse the development community is (contrary to common belief), it is natural that z2 has to fight with adoption. Specifically, the more profound critique towards z2 is about being too big, too proprietary, too non-standard.

So this is what version 3 is all about:

Less and more focussed

The one thing z2 is about is removing obstacles between code and execution. You should only think about code, modules, software structure. In order to enhance the “do one thing and do it well” qualities of z2, we will strip of capabilities that may not be totally obvious (like for example z2’s JTA implementation, support for worker processes) and either drop those completely or move them into addons.

Better and Friendlier Open Source

Z2 has always been open source. In version 3 all package names will be “org.z2env” and, possibly more interesting than that cosmetic change, we will make sure that there will be no use of libraries with a problematic license like GPL. Only Apache 2 or compatible licenses will be contained.

Integrating with Tomcat

Previously, z2 embedded Jetty as its preferred Web Container. Jetty is a great Web container and its embeddability is (rightfully) legendary. The vast majority of Java developers use Tomcat though.

With version 3 we found a cool way of hooking z2 up with your ordinary Tomcat installation and its configuration, so that Web applications defined in z2 work next to whatever else you have deployed.

If TomEE would not make such harsh structural assumptions on application deployment – assumptions we cannot agree with, and much less adhere to – we would even have EJBs in z2.

That is no big deal though – as I have the vague feeling that EJB enthusiasts are probably even less likely to adopt z2.

Getting Started

Enough talk! While there is still a lot to do (porting the Spring add-on and all the sample applications), a simple Getting Started guide can be found here


Willing to invest 15 Min into something cool? Here it is!

Feedback greatly welcome!

Map Reduce Applications with z2 – Part 2

A while back, actually quite some time back in Map Reduce Applications with z2 – Part 1, I promised some more details on how to run deeply integrated map-reduce jobs on Hadoop and HBase with z2.

I finally got around to describing a complete sample that I used for my talk Buildfrei Skalieren für Big Data mit Z2 (slides available following the link) at the Entwicklertag conference in Karlsruhe. It took me a year to finally turn this into a sample from the Wiki…

Anyway. You know Hadoop/HBase? You know how painful it can be to re-use application level services in mapper or reducer tasks? Then you should take a look at the

Sample that combines HBase with full-stack Spring and Hibernate usage


v2.3 is out

Now that it has been concluded (well, now that we decided) that Version 2.3 is what it is, here is some highlights:

A Z2 plugin for JetBrain’s IntelliJ IDEA

As with Eclipse it is now possible to have class path resolution and some more for Z2 in IntelliJ. IntelliJ’s approach to working with projects and modules (as opposed to workspaces and projects in Eclipse) is not-so-slightly different which strongly impacts what IDE-specific features make sense. For the Z2 plugin the one important additional feature is to discover Z2 modules and have them added and removed from the currently opened project at a click.

The plugin can be installed directly from the plugin repository operated by JetBrains.

Please read on at IntelliJ Z2 Plugin.

Maven repository access from Z2

In principle you wouldn’t want to use anything like an artifact repository when using Z2. Its component repositories serve (among other things) as a shareable source of binary artifacts and dependency information.

The true value in the combination of Z2 with Maven artifact repositories lies in the wealth of community-provided naming and dependency information plus the ability to start light when using an externally provided repository such as Maven Central. So the main benefit, a very real benefit, lies in its existence.

In version 2.3 it is possible to access Maven artifact repositories and consume artifacts. Add-ons as well as relevant samples have been ported.

Please read on at How to Acess a Maven Repository or better yet see things happening by running a simple sample such as the Sample-Spring-Basic.

As in all usages of artifact repositories, the dependency information provided does hardly ever say all you need to know in any but the most trivial cases. The March blog post Dependency Management for Modular Applications captures the theoretical side of assembly from artifact repositories for modular applications.

Java Language Level

Up to Version 2.2, Z2 was pretty much hard-coded to be all Java 6. That is, you needed a Java 6 Runtime Environment and all source code as well as byte code of binary artifacts where expected to be Java 6 compatible.

With Version 2.3, you can specify whether you want to target a Java 7 or a Java 6 environment and Z2 applies the natural default depending on what Java version it was started with.

Please read on at How to set the language level.

Dependency Management for Modular Applications

While working on a rather cool support for Maven Repositories in Z2 (all based on Aether – the original Maven dependency and repository implementation), I converted several scenarios to be using artifacts stored in Maven central. This little piece is on what I ran into.

It all starts with the most obvious of all questions:

How to best map Java components (Z2 term) from Maven artifacts?

As Java components (see here for more details) are the obvious way of breaking up stuff and sharing it at runtime, it is natural to turn every jar artifact into a Java component – exposing the actual library as its API. But then, given that we have so much dependency information, can we map that as well?

Can we derive class loader references from Maven dependencies?

Maven has different dependency scopes (see here). You have to make some choices when mapping to class loader references. As compile dependencies are just as directed as class loader references, mapping non-optional compile dependencies to class loader references between Java components seems to be a rather conservative and still useful approach.

At first glance this looks indeed useful. I was tempted to believe that a lot of real-world cases would fit. Of course there are those adapter libraries (like spring-context-support), that serve to integrate with a large number of other frameworks and therefore have tons of optional compile dependencies. Using them only with non-optional compile dependencies on the class path completely defeats their purpose. (For example spring-context-support is made to hook up your Spring application context with any subset of their target technologies – and only that needs to be on the effective class path.)

But there are other “bad” cases. Here are two important examples:

  • Consistency: The Hibernate O/R Mapper library integrates with the Java Transaction API (JTA) – and it definitely better should. So it does have a corresponding non-optional compile dependency on JTA, but not on javax.transaction:jta but rather JBOSS’s packaging of it. Spring-tx (the transaction utilities of the Spring framework) however has a compile dependency on javax.transaction:javax.transaction-api. Using both with the trivial mapping would hence lead to some rather unresolvable “type misunderstanding” a.k.a. class cast exception. So there is consistency problems around artifacts that have no clear ownership.
  • Runtime constraints: The spring-aspect library, that enables transparent, automatic Spring configuration for objects that are not instantiated by Spring, associates a Spring application context with a class loading namespace by holding on to an application context by a static class member. That is: Every “instance” of the spring-aspect library can be used with at most one application context. Hence, it makes no sense what so ever to share it.

So there are problems. Then again: Why split and share at runtime with class loader isolation anyway? Check the other article for some reasoning. Long story short: There is good cases for it (but it’s no free lunch).

Assuming we want to and considering the complexities above, we can conclude that we will see some non-trivial assembly.

Here are some cases I came up with:


  1. Direct mapping: Some cases allow to retain the compile dependency graph one to one as class loading references. This does however require to not rely on anything that is expected to be present on the platform. E. g. “provided” dependencies will not be fulfilled and most likely we end up in case 3.
  2. Isolated Inclusion: For the case of of spring-aspects, it assumes to be a class loading singleton. It is one for each application context. So, if you want to have multiple application contexts, you will need multiple copies of spring-aspects. This is true for all cases where some scope that can be instantiated multiple times is bound to some static class data.
  3. Aggregation: the not so rare case that a dependency graph has a single significant root and possibly some dependencies that should be excluded as they will be satisfied by the environment, it is most practical to form a new module by flattening the compile dependency graph (omitting all exclusions) and satisfy other runtime requirements via platform class loading means. The role model for this is Hibernate.
  4. Adapter Inclusion: Finally, we consider the case of a library that is made to be used in a variety of possible scopes. A great example is spring-context-support that provides adapters for using Spring with more other tools than you probably care in any single setup. If spring-context-support would be used via a classloading reference, it could not possibly be useful as it would not “see” the referencing module’s choice of other tools. By including (copying) it into the module, it can.


If you follow the “normal” procedure and simply assemble everything into one single web application class loader scope, you can spare some thinking effort and get around most of the problems outlined above. If you however have the need to modularize with isolated class loading hierarchies, you need to spend some time understanding toolsets and choosing an assembly approach. This does pay off however, as something that happened accidentally has now become a matter of reasoning.


  1. Z2 Maven Repo Support
  2. Z2 Java Components
  3. Maven Dependency Scopes


User Friendly Production Updates

February is almost over – so little time to spend on this blog. Here is a short post on something cool we did a while back in Z2.

The original request was along the lines of


Can we do an software upgrade without interrupting a user’s work?


This is in the context of the not so insignificant class of applications that require users to operate on some non-trivial but temporary and yet rich state before a persistent state change of the system can be performed. That is, applications that keep non-trivial session state where kicking out users means a real loss of time and nerves and is more than unfriendly.

Still – what if there is an important update to an intranet application that should be applied and, say, it should be tried by some group of users after lunch?

Applying a software upgrade to a running application without interfering with the work progress of currently logged-in users has some natural limitations. For example, smart data migrations will be extremely hard to get right (in a stateful scenario). But anything above the domain level might actually work.

Theoretically the most natural approach would be to temporarily store user session data somewhere, “replace” the application and load the user session data into memory again. Practically speaking however, it will be hard to find complex Java applications that use session serialization and would be assumed to reliably save and restore a user session. Furthermore, during the time of the application restart there would still be some downtime that may be taken as a system failure by users.

So, instead of doing something smart why not do something really obvious:


Leave the application running until the last user has logged off (or was logged off due to session expiration). Present the new application version to all user that log in after the update.


In fact, the approach we took leaves the whole (frontend) application stack running until the last session “running” has become invalid. It is implemented by the Gateway module and described in detail in the wiki. Here is a short summary:

Normally, a Z2 in “server mode” has at least two processes running: A home process that serves as a watch dog and synchronization service plus a webWorker node that runs the (Jetty) Web server and whatever Web applications are supposed to be up.

With the Gateway module this setup is altered (by configuration) in that the actual Web server entry point is now running in the home process and forwarding requests on a by-session scheme to the actual webWorker process. Worker processes, such as the webWorker, can now be detached from the synchronisation procedure. That is, instead of being stopped because of changes, detached worker processes are simply left unaffected from any updates until nobody needs them anymore:


Using that mechanism, users can decide to complete their current work and decide to upgrade at their convenience by logging off and on again.

Finally, this is another cool show case of how beneficial worker process control within the execution environment is.