Results for tag "batch"

Spring Batch as Wildfly Module

posted by Roberto Cortez on July 29, 2014

(TL;DR – Get me to the code)

For a long time, the Java EE specification was lacking a Batch Processing API. Today, this is an essential necessity for enterprise applications. This was finally fixed with the JSR-352 Batch Applications for the Java Platform now available in Java EE 7. The JSR-352 got it’s inspiration from the Spring Batch counterpart. Both cover the same concepts, although the resulting API’s are a bit different.

Since the Spring team also collaborated in the JSR-352, it was only a matter of time for them to provide an implementation based on Spring Batch. The latest major version of Spring Batch (version 3), now supports the JSR-352.

I’m a Spring Batch user for many years and I’ve always enjoyed that the technology had a interesting set of built-in readers and writers. These allowed you to perform the most common operations required by batch processing. Do you need to read data from a database? You could use JdbcCursorItemReader, how about writing data in a fixed format? Use FlatFileItemWriter, and so on.

Unfortunately, JSR-352 implementations do not have the amount of readers and writers available in Spring Batch. We have to remember that JSR-352 is very recent and didn’t have time to catch up. jBeret, the Wildfly implementation for JSR-352 already provides a few custom readers and writers.

What’s the point?

I was hoping that with the latest release, all the readers and writers from the original Spring Batch would be available as well. This is not the case yet, since it would take a lot of work, but there are plans to make them available in future versions. This would allow us to migrate native Spring Batch applications into JSR-352. We still have the issue of the implementation vendor lock-in, but it may be interesting in some cases.

Motivation

I’m one of the main test contributors for the Java EE Samples in the JSR-352 specification. I wanted to find out if the tests I’ve implemented have the same behaviour using the Spring Batch implementation. How can we do that?

Code

I think this exercise is not only interesting because of the original motivation, but it’s also useful to learn about modules and class loading on Wildfly. First we need to decide how are we going to deploy the needed Spring Batch dependencies. We could deploy them directly with the application, or use a Wildfly module. Modules have the advantage to be bundled directly into the application server and can be reused by all deployed applications.

Adding Wildfly Module with Maven

With a bit of work it’s possible to add the module automatically with the Wildfly Maven Plugin and the CLI (command line). Let’s start to create two files that represent the CLI commands that we need to create and remove the module:

wildfly-add-spring-batch.cli

# Connect to Wildfly instance
connect

# Create Spring Batch Module
# If the module already exists, Wildfly will output a message saying that the module already exists and the script exits.
module add \
    --name=org.springframework.batch \
    --dependencies=javax.api,javaee.api \
    --resources=${wildfly.module.classpath}

# Connect to Wildfly instance

connect

# Create Spring Batch Module

# If the module already exists, Wildfly will output a message saying that the module already exists and the script exits.

module add \

--name=org.springframework.batch \

--dependencies=javax.api,javaee.api \

--resources=${wildfly.module.classpath}

The module --name is important. We’re going to need it to reference it in our application. The --resources is a pain, since you need to indicate a full classpath to all the required module dependencies, but we’re generating the paths in the next few steps.

wildfly-remove-spring-batch.cli

# Connect to Wildfly instance
connect

# Remove Oracle JDBC Driver Module
module remove --name=org.springframework.batch

# Connect to Wildfly instance

connect

# Remove Oracle JDBC Driver Module

module remove --name=org.springframework.batch

Note wildfly.module.classpath. This property will hold the complete classpath for the required Spring Batch dependencies. We can generate it with Maven Dependency plugin:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <version>${version.plugin.dependency}</version>
    <executions>
        <execution>
            <phase>generate-sources</phase>
            <goals>
                <goal>build-classpath</goal>
            </goals>
            <configuration>
                <outputProperty>wildfly.module.classpath</outputProperty>
                <pathSeparator>:</pathSeparator>
                <excludeGroupIds>javax</excludeGroupIds>
                <excludeScope>test</excludeScope>
                <includeScope>provided</includeScope>
            </configuration>
        </execution>
    </executions>
</plugin>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-dependency-plugin</artifactId>

<version>${version.plugin.dependency}</version>

<phase>generate-sources</phase>

<goals>

<goal>build-classpath</goal>

</goals>

<outputProperty>wildfly.module.classpath</outputProperty>

<excludeGroupIds>javax</excludeGroupIds>

<includeScope>provided</includeScope>

</configuration>

</execution>

</executions>

</plugin>

This is going to pick all dependencies (including transitive), exclude javax (since they are already present in Wildfly) and exclude test scope dependencies. We need the following dependencies for Spring Batch:

<!-- Needed for Wildfly module -->
<dependency>
    <groupId>org.springframework.batch</groupId>
    <artifactId>spring-batch-core</artifactId>
    <version>3.0.0.RELEASE</version>
    <scope>provided</scope>
</dependency>

<dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring-jdbc</artifactId>
    <version>4.0.5.RELEASE</version>
    <scope>provided</scope>
</dependency>

<dependency>
    <groupId>commons-dbcp</groupId>
    <artifactId>commons-dbcp</artifactId>
    <version>1.4</version>
    <scope>provided</scope>
</dependency>

<dependency>
    <groupId>org.hsqldb</groupId>
    <artifactId>hsqldb</artifactId>
    <version>2.3.2</version>
    <scope>provided</scope>
</dependency>

<groupId>org.springframework.batch</groupId>

<artifactId>spring-batch-core</artifactId>

<version>3.0.0.RELEASE</version>

<scope>provided</scope>

</dependency>

<groupId>org.springframework</groupId>

<artifactId>spring-jdbc</artifactId>

<version>4.0.5.RELEASE</version>

<scope>provided</scope>

</dependency>

<groupId>commons-dbcp</groupId>

<artifactId>commons-dbcp</artifactId>

<scope>provided</scope>

</dependency>

<groupId>org.hsqldb</groupId>

<artifactId>hsqldb</artifactId>

<scope>provided</scope>

</dependency>

Now, we need to replace the property in the file. Let’s use Maven Resources plugin:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-resources-plugin</artifactId>
    <version>${version.plugin.resources}</version>
    <executions>
        <execution>
            <id>copy-resources</id>
            <phase>process-resources</phase>
            <goals>
                <goal>copy-resources</goal>
            </goals>
            <configuration>
                <outputDirectory>${basedir}/target/scripts</outputDirectory>
                <resources>
                    <resource>
                        <directory>src/main/resources/scripts</directory>
                        <filtering>true</filtering>
                    </resource>
                </resources>
            </configuration>
        </execution>
    </executions>
</plugin>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-resources-plugin</artifactId>

<version>${version.plugin.resources}</version>

<id>copy-resources</id>

<phase>process-resources</phase>

<goals>

<goal>copy-resources</goal>

</goals>

<outputDirectory>${basedir}/target/scripts</outputDirectory>

<directory>src/main/resources/scripts</directory>

</resource>

</resources>

</configuration>

</execution>

</executions>

</plugin>

This will filter the configured files and replace the property wildfly.module.classpath with the value we generated previously. This is a classpath pointing to the dependencies in your local Maven repository. Now with Wildfly Maven Plugin we can execute this script (you need to have Wildfly running):

<plugin>
    <groupId>org.wildfly.plugins</groupId>
    <artifactId>wildfly-maven-plugin</artifactId>
    <version>${version.plugin.wildfly}</version>
    <configuration>
        <skip>false</skip>
        <executeCommands>
            <batch>false</batch>
            <scripts>
                <!--suppress MavenModelInspection -->
                <script>target/scripts/${cli.file}</script>
            </scripts>
        </executeCommands>
    </configuration>
</plugin>

<groupId>org.wildfly.plugins</groupId>

<artifactId>wildfly-maven-plugin</artifactId>

<version>${version.plugin.wildfly}</version>

<skip>false</skip>

<batch>false</batch>

</scripts>

</executeCommands>

</configuration>

</plugin>

And these profiles:

<profiles>
    <profile>
        <id>install-spring-batch</id>
        <properties>
            <cli.file>wildfly-add-spring-batch.cli</cli.file>
        </properties>
    </profile>

    <profile>
        <id>remove-spring-batch</id>
        <properties>
            <cli.file>wildfly-remove-spring-batch.cli</cli.file>
        </properties>
    </profile>
</profiles>

<id>install-spring-batch</id>

<cli.file>wildfly-add-spring-batch.cli</cli.file>

</properties>

</profile>

<id>remove-spring-batch</id>

<cli.file>wildfly-remove-spring-batch.cli</cli.file>

</properties>

</profile>

</profiles>

(For the full pom.xml contents, check here)

We can add the module by executing:
mvn process-resources wildfly:execute-commands -P install-spring-batch.

Or remove the module by executing:
mvn wildfly:execute-commands -P remove-spring-batch.

This strategy works for any module that you want to create into Wildfly. Think about adding a JDBC driver. You usually use a module to add it into the server, but all the documentation I’ve found about this is always a manual process. This works great for CI builds, so you can have everything you need to setup your environment.

Use Spring-Batch

Ok, I have my module there, but how can I instruct Wildfly to use it instead of jBeret? We need to add a the following file in META-INF folder of our application:

jboss-deployment-structure.xml

<?xml version="1.0" encoding="UTF-8"?>
<jboss-deployment-structure>
    <deployment>
        <exclusions>
            <module name="org.wildfly.jberet"/>
            <module name="org.jberet.jberet-core"/>
        </exclusions>

        <dependencies>
            <module name="org.springframework.batch" services="import" meta-inf="import"/>
        </dependencies>
    </deployment>
</jboss-deployment-structure>

<?xml version="1.0" encoding="UTF-8"?>

<jboss-deployment-structure>

</exclusions>

</dependencies>

</deployment>

</jboss-deployment-structure>

Since the JSR-352 uses a Service Loader to load the implementation, the only possible outcome would be to load the service specified in org.springframework.batch module. Your batch code will now run with the Spring Batch implementation.

Testing

The github repository code, has Arquillian sample tests that demonstrate the behaviour. Check the Resources section below.

Resources

You can clone a full working copy from my github repository. You can find instructions there to deploy it.

Wildfly – Spring Batch

Since I may modify the code in the future, you can download the original source of this post from the release 1.0. In alternative, clone the repo, and checkout the tag from release 1.0 with the following command: git checkout 1.0.

Future

I’ve still need to apply this to the Java EE Samples. It’s on my TODO list.

Speaking at JavaOne 2014

posted by Roberto Cortez on June 11, 2014

8 Comments

tags: batch, conference, horror story, javaee, javaone, legacy code, speaker

Yesteday I got really great news. I was selected to present 3 out of 4 sessions that I have submitted to JavaOne 2014! After attending the first JavaOne in 2012, and going again in 2013, this was my first time submitting something for JavaOne.

I have to confess that I had high hopes of being selected, but I was not expecting to have 3 sessions right in the first year of submissions, since there are a lot of submissions and it’s really hard to get selected. A special thanks to Reza Rahman for helping me out during the submission process and for providing valuable tips. Thanks Reza! I would also like to thank you Ivan Ivanov and Simon Maple my co-speakers in two of the sessions.

Have a look below into the sessions abstracts and videos. I don’t have the schedules yet, but look for them in the JavaOne Schedule Builder (when available) and signup 🙂

What am I going to speak about?

CON2818 – Java EE 7 Batch Processing in the Real World

with Ivan Ivanov

Abstract
This talk will explore one of the newest API for Java EE 7, the JSR 352, Batch Applications for the Java Platform. Batch processing is found in nearly every industry when you need to execute a non-interactive, bulk-oriented and long running operation task. A few examples are: financial transactions, billing, inventory management, report generation and so on. The JSR 352 specifies a common set of requirements that every batch application usually needs like: checkpointing, parallelization, splitting and logging. It also provides you with a job specification language and several interfaces that allow you to implement your business logic and interact with the batch container. We are going to live code a real life example batch application, starting with a simple task and then evolve it using the advanced API’s until we have a full parallel and checkpointing reader-processor-writer batch. By the end of the session, attendees should be able to understand the use cases of the JSR 352, when to apply it and how to develop a full Java EE Batch Application.

BOF4223 – Development Horror Stories

with Simon Maple

Abstract
We all enjoy to hear a good success story, but in the software development industry the life of a developer is also made up of disasters, disappointments and frustrations. Have you ever deleted all the data in production? Or maybe you just run out of disk space and your software failed miserably! How about crashing your server with a bug that you introduced in the latest release? We can learn with each others with the mistakes we made. Come to this BOF and share with us your most horrific development story and what did you do to fix it.

CON4255 – The 5 people in your organization that grow legacy code

Abstract
Have you ever looked at a random piece of code and wanted to rewrite it so badly? It’s natural to have legacy code in your application at some point. It’s something that you need to accept and learn to live with. So is this a lost cause? Should we just throw in the towel and give up? Hell no! Over the years, I learned to identify 5 main creators/enablers of legacy code on the engineering side, which I’m sharing here with you using real development stories (with a little humour in the mix). Learn to keep them in line and your code will live longer!