Z2-environment Version 2.6 is ready – for download!

Finally. I am happy to declare version 2.6 ready for download and use.

Version 2.6 comes with a lot of small improvements and some due follow up on what changed in the Java world.

Aside from regular software maintenance, there is one bigger change: The z2-base distribution.

If you are wondering what you might be missing, read on!

Check out the updated wiki and online documentation.

Java 9 and Java 10 Support

One of the more obvious changes is that v2.6 requires Java 9 and runs with Java 9, 10, and with Java 11 as well. Language-wise, this release is on Java 9 or Java 10 depending on the runtime used.

Java 9 introduced a new module system feature into the Java core model (see the Jigsaw Project). Unfortunately this system has even more flaws than OSGi had as far as considering its usefulness for actually application development. Lets put it this way: You will absolutely not be bothered with Java 9 modularity when using Z2 modularity.

Z2 Distribution for Download

Beyond many useful upgrades and minor improvements, some shown below, the one important “innovation” is that z2 now has a download page and a downloadable distribution.  One of Z2’s defining features is to pull from local and online resources and prepare anything required for running all by itself. That philosophy was very visible in the way we promoted the use of Z2 previously: Only checkout the core and pull anything else from (our) repositories. Problematic about that approach is that it makes creating your own system a unnecessarily complicated procedure. Providing a distribution simplifies getting started on your own and gives you a clean set of assets to import. Last but not least, we provide you with a comprehensive overview over all included 3rd-party licenses:

More Desktop for the GUI

The Z2 GUI that is really just a log container with buttons for the few interactions you need with Z2 is now a little more friendly to the eye by offering font scaling by pressing Ctrl and htting “+” or “-” or turning the mouse wheel.

Expert Features

Linked Components

Link component work like symbolic file system links in that they move the visibility of a component definition to a different module and component name. The link may actually add additional information, such as dependencies, state participation and more. (documentation)

Parameterized tests and suites in z2Unit

Z2Unit, the integrated JUnit-based testing kit for seamless as-easy-as-ever in-container testing had some gaps due some omissions in JUnit’s internal APIs. Z2Unit strives to move the convenience of a local class test to a deep integrated, painless in-system testing. (blog)

Finer Control on Compile Order and Source-Jar filtering

Some optimizations have been added to provide finer control on use of extension compilers (e.g. for AspectJ) for single Java component facets (API, impl, test). Previously z2 also made source JAR files visible to the application. This was neat for development but expensive and in some cases outright problematic at runtime. (documentation)

Clean Up of Third Party Libraries and Various Upgrades

Z2 now integrates Jetty 9.4.8. and JTA 1.2 for its built-in transaction support. Samples and sample dependencies have generally be upgraded to recent versions of e.g. Spring and Hibernate. Check out the version page.

Advertisements
Posted in z2

Java 9 Module System – Useful or Not?

Actually… rather not.

I am currently working on preparing z2-Environment version 2.6. While it is not used by a lot of teams, where it is used we have large, modular, code bases.

This blog is packed with posts on all kinds of aspects of modularization of largish software solutions. Essentially it all boils down to isolation, encapsulation, and sharing to keep complexity under control by separating inner complexities from the “public model” of a solution, in order to foster long-term maintainability and ability to change and extend.

Modularization is a means of preserving structural sanity and comprehensibility over time.

That said, modularization is a concern of developing and evolving software solutions – not libraries.

The Java 9 module system is however exactly that: A means to express relationships between JAR libraries within JAR libraries. I wrote about it before a while back (Java Modularity – Failing once more?). Looking at it again – I found nothing that makes it look more useful.

First of all very few libraries out there will have useful module descriptors – unless they work together trivially anyway. Inconsistent Maven dependencies are bad enough, but can usually worked around in your own assembly. A bad or missing module descriptor essentially requires you to change an existing library.

Even, if all was good, clean, consistent with our popular third-party libraries, what problem would those module-infos and the module system actually solve for us?

The only effective means to really hide implementation details to the extent of keeping definitions completely out of visibility, the means to very controlled expose definitions, even if other versions of identically-named definitions are present in the system is still a class loader based modularization approach. Java 9 modularization does not preclude that. It does not add anything useful either as far as I can tell.

Z2 v2.9 will not have explicit integration for Java 9 module system for now – for lack of usefulness.

References

Z2-environment Version 2.5 Is Out

It took a while, but it got finally done. Version 2.5 of the z2-environment is out. Documentation and samples have been updated and tested.

Here is what version 2.5 was about:

More Flexibility in Component Configuration

A major feature of z2 is to run a system cluster strictly defined by centrally, version-controlled configuration. As there is no rule without an exception, some configuration is just better defined by the less static and local system runtime environment, such as environment variable or scripted system properties.

To support that better and without programming, component properties may now be expressed by an extensible expression evaluation scheme with built-in JEXL support. Expressions are evaluated upon first load of component descriptors after start or after invalidation at runtime.

Some use-cases are:

  • Seamless branch or URL switching based on environment settings.
  • Dynamic evaluation of database config, in particular remote authentication based on custom evaluators or environment settings.
  • Declarative sharing of configuration across components.

Some aspects, such dynamic evaluation of searchable properties, were not made dynamic due to the risk of unpredictable behavior. Future work may show that the concept can be extended further though.

pseudo_uml

Check it out in the documentation.

More Complete in-Container Test Support

Z2 offers a sleek, built-in way of running application-side in-container tests: z2Unit. Previously, the JUnit API had its limits in serializabilty over the wire – which is essential for z2Unit. JUnit improved greatly in that department and after the correspinding adaptation of z2Unit some problematic Unit Test Runner combinations (such as e.g. z2Unit and parameterized tests) now work smoothly.

z2unit

Check it out in the documentation.

Better Windows Support

Some very old problems with blanks in path or parameter names got finally fixed. There is a straight forwared command line specification syntax for worker processes that is (mostly) backward compatible.

Also, and possibly more importantly, system property propagation from Home to Worker processes is now fully configurable.

Check it out in the documentation.

Better Git Support

Z2 can read directly from GIT repositories. However, previously only a branch could be specified as content selector. Now any GIT ref will do.

thatsit

Check it out in the documentation.

There is some more. Please check out the version page, if you care.

What’s next:

The plans for 2.6 are still somewhat open. As the work in Version 3 will not make it into any production version – namespace changes are too harming at this time – some useful structural simplifications implemented in 3.0 are considered, such as:

  • Worker processes as participants in Home process target states (rather than a Home Layout)
  • Introducing a “root” repository that hosts any number of remote or “inline” repositories and so streamlines local core config
  • Supporting a co-located Tomcat Web Container as an alternative to an integrated Jetty Web Container
  • Component templates that provide declarative default configurations and so remove duplications (i.e. any Java component based on Spring+Spring AOP).

Thanks and good luck an whatever you do that needs to be done right!

References

 

 

Posted in z2

Java Modularity – Failing once more?

Like so many others, I have pretty much ignored project Jigsaw for some time now – assuming it would stay irrelevant for my work or slowly fade away and be gone for good. The repeated shifts in planned inclusion with the JDK seemed to confirm this course. Jigsaw started in 2009 – more than six years ago.

Jigsaw is about establishing a Java Module System deeply integrated with the Java language and core Java runtime specifications. Check out its goals on the project home page. It is important to note the fourth goal:

 

Make it easier for developers to construct and maintain libraries and large applications, for both the Java SE and EE Platforms.

 

Something Missing?

Lately I have run into this mail thread: http://permalink.gmane.org/gmane.comp.java.openjdk.jigsaw/2492

In that mail thread Jürgen Höller (of Spring fame) notes that in order to map Spring’s module layout to Jigsaw modules, it would be required to support optional dependencies – dependencies that may or may not be satisfied given the presence of another module at runtime.

This is how Spring supports its set of adapter and support types for numerous other frameworks that you may or may not use in your application: Making use of Java’s late linking approach, it is possible to expose types that may not be usable without the presence of some dependent type but will not create a problem unless you start using them either. That is, optional dependencies would allow Spring to preserve its way of encapsulating the subject of „Spring support for many other libraries“ into one single module (or actually a jar file).

In case you do not understand the technical problem, it is sufficient to note that anybody who has been anywhere near Java class loading considerations as well as actual Java application construction in real live should know that Spring’s approach is absolutely common for Java infrastructure frameworks.

Do Jigsaw developers actually know or care about Java applications?

Who knows, maybe they simply forgot to fix their goals. I doubt it.

 

Module != JAR file

 

There is a deeper problem: The overloaded use of the term module and the believe of infrastructure developers in the magic of the term.

Considering use of the module term in programming languages, it typically denotes some encapsulation of code with some interface and rules on how to expose or require some other module. This is what Jigsaw focussed on and it is what OSGi focussed on. It is what somebody interested in programming language design would most likely do.

In Java this approach naturally leads using or extending the class loading mechanism to expose or hide types between modules for re-use (or information hiding resp.) which in turn means to invent descriptors that describe use relationships (meaning the ability to reference types in this case) and so on.

This is what Jigsaw does and this is what OSGi did for that matter.

It is not what application developers care about – most of the time.

There is an overlap in interest of course. Code modules are an important ingredient in application assembly. Problems of duplicate type definitions by the same name (think different library versions) and separation of API and implementation are essential to scalable, modular system design.

But knowing how to build a great wall is not the same as knowing how to build a great house.

From an application development perspective, a module is much rather a generic complexity management construct. A module encapsulates a responsibility and in particular should absolutely not be limited to code and is not particularly well-served by squeezing everything into the JAR form factor.

What we see here is a case of Application vs. Infrastructure culture clash in action (see for example Local vs. Distributed Complexity).

The focus on trying to find a particularly smart and technically elegant solution for the low-level modularization problem eventually hurts the usefulness of the result for the broader application development community (*).

Similarly, ignorance of runtime modularization leads to unmaintainable, growth-limited, badly deployable code bases as I tried to describe in Modularization is more than cutting it into pieces.

The truth is somewhere in between – which is necessarily less easily described and less universal in nature.

I believe that z2 is one suitable approach for a wide class of server-side applications. Other usage scenarios might demand other approaches.

I believe that Jigsaw will not deliver anything useful for application developers.

I wish you a happy new year 2016!

References

Ps.:

* One way of telling that the approach will be useless for developers is when discussions conclude that “tools will fix the complexity”. What that comes down to is that you need a compiler (the tool) to make use of the feature, which in turn means you need another input language. So who is going to design that language and why would that be easier?

* It is interesting to check out the history of SpringSource’s dm Server (later passed on to R.I.P. at Eclipse as project Virgo). See in particular the interview with Rod Johnson.

Z2 as a Functional Application Server

Intro

As promised this is first in a series of posts elaborating on integration of Clojure with Z2. It probably looks like a strange mix, however I believe it’s extremely empowering combination of two technologies sharing lots of design philosophy. Clojure has brought me lots of joy by enabling me to achieve much more in my hobby projects and I see how the combination of z2 and Clojure further extends the horizon of what’s possible. I’d be happy if I manage to help other people give it a try and benefit in the same way I did.

The LISP universe is very different one. It’s hard to convince someone with zero previous experience to look at the strange thing with tons of parentheses written using polish notation, so I am going to share my personal story and hope it resonates with you. I will focus on the experience or how I “felt” about it. There is enough theory and intellectual knowledge on the internet already and I will link to it where appropriate.

So, given this is clearly personal and subjective view, let’s put in some context.

Short Bio

I’ve been using Java professionally for 12+ years. Predominantly in the backend. I’ve worked on application servers, as well as on business applications in large, medium and small organizations. Spring is also something I have been heavily relying on in the last 8 years. Same goes for maven. I’ve used JBoss and done bit of application server development myself, but when Spring Boot came up I fell in love with it.

Like every other engineer out there my major struggle through the years have been to manage the complexity. The inherent complexity of the business problem we have to solve plus the accidental complexity added by our tools, our poor understanding of the problem domain and our limited conceptual vocabulary. I have been crushed more than once under the weight of that complexity. Often my own and the team’s share would be more than 50%. I have seen firsthand how poorly groomed code base ends up in state where the next feature is just not possible. This has real business impact.

The most scary thing about complexity is that it grows exponentially with size. This is why I strongly subscribe to the “code is liability” worldview. Same goes for organizations. The slimmer you are the faster and further you can go.

Ways to deal with complexity

Now that the antagonist is clearly labeled, let’s focus on my survival kit.

#1 Modularization

One powerful way to get on top of complexity is divide and conquer by using modularization. This is where z2 comes into the game. It has other benefits as well, but I would put it’s modularization capabilities as feature #1. Maven and Spring have been doing that for me through the years. On more coarse level Tomcat and JBoss provide some modularization facilities as well, however it is extremely rare in my experience where they are deliberately exploited.

Getting modularization right is hard on both ends:

  • The framework has to strike a balance between exercising control and enabling extensibility, otherwise it becomes impractical.
  • The component developers still have to think hard and define boundaries of “things” while using the framework idioms with mastery. I haven’t met yet technology that removes this need. It’s all about methodology and concepts (I dislike the pattern cult).

Discussing more precise definition of what exactly is modularization and why the well-known methodologies are too general to be useless as recipes is too big of discussion for here.

My claim is that z2 strikes the best balance I have seen so far while employing very powerful concept.

#2 Abstractions

Another powerful way is to use better abstractions. While modularization puts structure in chaos, the right abstractions reduce the amount of code and other artifacts, hence the potential for chaos. Just like any other thing, all abstractions are not made equal and I assume they can be ordered according to their power.

My personal definition for power: if abstraction A allows you to achieve the same result with less code than abstraction B then it’s more powerful. Of course reality is much more hairy than this. You have to account for abstraction’s implementation, investment in learning, long term maintenance costs & so on.

Alternative definition: if abstraction A allows you to get further in terms of project size and complexity (before the project collapses) it’s more powerful.

The abstractions we use on daily basis are strongly influenced by the language. Language can encourage, discourage (due ergonomics) or even blacklist an abstraction by having no native support for it. My claim here is that the Java language designers have made some very limiting choices and this has profound effect on the overall productivity as well as the potential access to new abstractions. Clojure, on the other side has excellent mix right out of the box with convenient access to very wide range of other abstractions.

The OO vs. FP discussion deserves special attention and will get it. I won’t claim that Clojure is perfect, far from it. However the difference in power I have experienced is significant and big part of that difference is due to carefully picked set of base abstractions implemented in very pragmatic way.

 So, what’s next?

Next comes the story how Java and DDD helped me survive and how JavaScript made me feel like a fool for wasting so many hours slicing problems the wrong way and worrying about insignificant things. Clojure will show up as well, you can count on this.

While you wait for the next portion, here are two links that have influenced heavily my current thinking:

  • Beating the averages — the blub paradox has been an eye opening concept for me. I have read this article in 2008 for the first time and kept coming back to it. It validated my innate tendency to be constantly dissatisfied with how things are and look for something better. Paradoxically, it never made me try out LISP 🙂
  • Simple made easy — This is the presentation that among other side effects made me give Clojure a chance. This presentation probably has the best return of investment for an hour spent in front of the screen.

Continuity is King

towerUnfortunately there has been so much going on in my work life and my private life lately that I didn’t get around thinking and writing much.

Here is just a short note that v2.4 of z2 is ready: v2.4

It simply upgrades z2 to Java 8 and upgrades the version of Jetty we use to 9.3. The latter implies that Java 8 is a strict requirement too.

Little change is good as those projects that run with z2 need anything but platform disruption at this time.

As the core did not change incompatibly (at least not that I know), using a v2.4 core with previous z2-base version 2.3 will simply add Java 8 support to any such setup as well.

Anyway here’s what I hope to continue on once I am back to normal operations again:

  • A piece on how to use clojure on z2 with a guest author from Vienna
  • A piece on a variation of the Stockholm Syndrome that can be observed in the relationship of developers with their toolset
  • A piece on how organization can be classified as Project vs. Product driven

 

Microservices Nonsense

Microservice Architecture (MSA) is a software design approach in which applications are intentionally broken up into remoteable services, in order built from small and independently deployable application building blocks with the goal of reducing deployment operations and dependency management complexity.

(See also Fowler, Thoughtworks)

Back in control

Sounds good, right? Anybody developing applications of some size knows that increasing complexity leads to harder to manage updates, increased deployment and restart durations, more painful distribution of deployables. In particular library dependencies have the tendency to get out of control and version graphs tend to become unmanageable.

So, why not break things up into smaller pieces and gain back control?

This post is on why that is typically the wrong conclusion and why Microservice Architecture is a misleading idea.

Conjunction Fallacy

From a positive angle one might say that MSA is a harmless case of a conjunction fallacy: Because the clear cut, that sounds more specific as a solution approach, makes it more plausible (see the Linda Problem of ….).

If you cannot handle it here, why do you think you can handle it here?

If you cannot organize your design to manage complexity in-process however, why should things work out more smoothly, if you move to a distributed setup where aspects like security, transaction boundaries, interface compatibility, and modifiability are substantially harder to manage (see also Distributed big ball of… )

No question, there can be good reasons for distributed architectures: Organization, load distribution, legacy systems, different expertise and technology preferences.

It’s just the platform (and a little bit of discipline)

Do size of deployables and dependency management complexity belong into that list?

No. The former simply implies that your technology choice has a poor roll-out model. In particular Java EE implementations are notoriously bad at handling large code bases (unlike, you might have guessed, z2). Similarly, loss of control over dependencies shows a lack of dependency discipline and, more often, a brutal lack of modularization effort and capabilities (see also Modularization is….)

Use the right tool

Now these problems might lead to an MSA approach out of desperation. But one should at least be aware that this is platform short-coming and not a logical implication of functional complexity.

If you were asked to move a piece of furniture you would probably use your car. If you were asked to move ten pieces of furniture, you would not look for ten cars – you would get a truck.

 

 

Working on z2env Version 3

Despite its fantastic qualities as a development and execution environment, z2’s adoption is very low. That of course does not at all stop us from improving it further (as we are actively benefiting from it anyway).

Whenever I talk about z2, the feedback is typically in one of two categories.

The first one is the “I don’t get it”-category.

There was a time when running builds was such a natural ingredient of software development to me, that I would have been in that category as well. So I forgive them their ignorance.

The other category is the “Great idea – … to bad I cannot use it”-category.

Being a disruptive approach and knowing how change averse the development community is (contrary to common belief), it is natural that z2 has to fight with adoption. Specifically, the more profound critique towards z2 is about being too big, too proprietary, too non-standard.

So this is what version 3 is all about:

Less and more focussed

The one thing z2 is about is removing obstacles between code and execution. You should only think about code, modules, software structure. In order to enhance the “do one thing and do it well” qualities of z2, we will strip of capabilities that may not be totally obvious (like for example z2’s JTA implementation, support for worker processes) and either drop those completely or move them into addons.

Better and Friendlier Open Source

Z2 has always been open source. In version 3 all package names will be “org.z2env” and, possibly more interesting than that cosmetic change, we will make sure that there will be no use of libraries with a problematic license like GPL. Only Apache 2 or compatible licenses will be contained.

Integrating with Tomcat

Previously, z2 embedded Jetty as its preferred Web Container. Jetty is a great Web container and its embeddability is (rightfully) legendary. The vast majority of Java developers use Tomcat though.

With version 3 we found a cool way of hooking z2 up with your ordinary Tomcat installation and its configuration, so that Web applications defined in z2 work next to whatever else you have deployed.

If TomEE would not make such harsh structural assumptions on application deployment – assumptions we cannot agree with, and much less adhere to – we would even have EJBs in z2.

That is no big deal though – as I have the vague feeling that EJB enthusiasts are probably even less likely to adopt z2.

Getting Started

Enough talk! While there is still a lot to do (porting the Spring add-on and all the sample applications), a simple Getting Started guide can be found here

https://redmine.z2-environment.net/projects/z2env/wiki/Getting_Started

Willing to invest 15 Min into something cool? Here it is!

Feedback greatly welcome!

Posted in z2

Map Reduce Applications with z2 – Part 2

A while back, actually quite some time back in Map Reduce Applications with z2 – Part 1, I promised some more details on how to run deeply integrated map-reduce jobs on Hadoop and HBase with z2.

I finally got around to describing a complete sample that I used for my talk Buildfrei Skalieren für Big Data mit Z2 (slides available following the link) at the Entwicklertag conference in Karlsruhe. It took me a year to finally turn this into a sample from the Wiki…

Anyway. You know Hadoop/HBase? You know how painful it can be to re-use application level services in mapper or reducer tasks? Then you should take a look at the

Sample that combines HBase with full-stack Spring and Hibernate usage

modules

Posted in z2

v2.3 is out

Now that it has been concluded (well, now that we decided) that Version 2.3 is what it is, here is some highlights:

A Z2 plugin for JetBrain’s IntelliJ IDEA

As with Eclipse it is now possible to have class path resolution and some more for Z2 in IntelliJ. IntelliJ’s approach to working with projects and modules (as opposed to workspaces and projects in Eclipse) is not-so-slightly different which strongly impacts what IDE-specific features make sense. For the Z2 plugin the one important additional feature is to discover Z2 modules and have them added and removed from the currently opened project at a click.

The plugin can be installed directly from the plugin repository operated by JetBrains.

Please read on at IntelliJ Z2 Plugin.

Maven repository access from Z2

In principle you wouldn’t want to use anything like an artifact repository when using Z2. Its component repositories serve (among other things) as a shareable source of binary artifacts and dependency information.

The true value in the combination of Z2 with Maven artifact repositories lies in the wealth of community-provided naming and dependency information plus the ability to start light when using an externally provided repository such as Maven Central. So the main benefit, a very real benefit, lies in its existence.

In version 2.3 it is possible to access Maven artifact repositories and consume artifacts. Add-ons as well as relevant samples have been ported.

Please read on at How to Acess a Maven Repository or better yet see things happening by running a simple sample such as the Sample-Spring-Basic.

As in all usages of artifact repositories, the dependency information provided does hardly ever say all you need to know in any but the most trivial cases. The March blog post Dependency Management for Modular Applications captures the theoretical side of assembly from artifact repositories for modular applications.

Java Language Level

Up to Version 2.2, Z2 was pretty much hard-coded to be all Java 6. That is, you needed a Java 6 Runtime Environment and all source code as well as byte code of binary artifacts where expected to be Java 6 compatible.

With Version 2.3, you can specify whether you want to target a Java 7 or a Java 6 environment and Z2 applies the natural default depending on what Java version it was started with.

Please read on at How to set the language level.

Posted in z2