Spring Batch as Wildfly Module

posted by Roberto Cortez on July 29, 2014

(TL;DR – Get me to the code)

For a long time, the Java EE specification was lacking a Batch Processing API. Today, this is an essential necessity for enterprise applications. This was finally fixed with the JSR-352 Batch Applications for the Java Platform now available in Java EE 7. The JSR-352 got it’s inspiration from the Spring Batch counterpart. Both cover the same concepts, although the resulting API’s are a bit different.

Since the Spring team also collaborated in the JSR-352, it was only a matter of time for them to provide an implementation based on Spring Batch. The latest major version of Spring Batch (version 3), now supports the JSR-352.

I’m a Spring Batch user for many years and I’ve always enjoyed that the technology had a interesting set of built-in readers and writers. These allowed you to perform the most common operations required by batch processing. Do you need to read data from a database? You could use JdbcCursorItemReader, how about writing data in a fixed format? Use FlatFileItemWriter, and so on.

Unfortunately, JSR-352 implementations do not have the amount of readers and writers available in Spring Batch. We have to remember that JSR-352 is very recent and didn’t have time to catch up. jBeret, the Wildfly implementation for JSR-352 already provides a few custom readers and writers.

What’s the point?

I was hoping that with the latest release, all the readers and writers from the original Spring Batch would be available as well. This is not the case yet, since it would take a lot of work, but there are plans to make them available in future versions. This would allow us to migrate native Spring Batch applications into JSR-352. We still have the issue of the implementation vendor lock-in, but it may be interesting in some cases.

Motivation

I’m one of the main test contributors for the Java EE Samples in the JSR-352 specification. I wanted to find out if the tests I’ve implemented have the same behaviour using the Spring Batch implementation. How can we do that?

Code

I think this exercise is not only interesting because of the original motivation, but it’s also useful to learn about modules and class loading on Wildfly. First we need to decide how are we going to deploy the needed Spring Batch dependencies. We could deploy them directly with the application, or use a Wildfly module. Modules have the advantage to be bundled directly into the application server and can be reused by all deployed applications.

Adding Wildfly Module with Maven

With a bit of work it’s possible to add the module automatically with the Wildfly Maven Plugin and the CLI (command line). Let’s start to create two files that represent the CLI commands that we need to create and remove the module:

wildfly-add-spring-batch.cli

# Connect to Wildfly instance
connect

# Create Spring Batch Module
# If the module already exists, Wildfly will output a message saying that the module already exists and the script exits.
module add \
    --name=org.springframework.batch \
    --dependencies=javax.api,javaee.api \
    --resources=${wildfly.module.classpath}

# Connect to Wildfly instance

connect

# Create Spring Batch Module

# If the module already exists, Wildfly will output a message saying that the module already exists and the script exits.

module add \

--name=org.springframework.batch \

--dependencies=javax.api,javaee.api \

--resources=${wildfly.module.classpath}

The module --name is important. We’re going to need it to reference it in our application. The --resources is a pain, since you need to indicate a full classpath to all the required module dependencies, but we’re generating the paths in the next few steps.

wildfly-remove-spring-batch.cli

# Connect to Wildfly instance
connect

# Remove Oracle JDBC Driver Module
module remove --name=org.springframework.batch

# Connect to Wildfly instance

connect

# Remove Oracle JDBC Driver Module

module remove --name=org.springframework.batch

Note wildfly.module.classpath. This property will hold the complete classpath for the required Spring Batch dependencies. We can generate it with Maven Dependency plugin:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <version>${version.plugin.dependency}</version>
    <executions>
        <execution>
            <phase>generate-sources</phase>
            <goals>
                <goal>build-classpath</goal>
            </goals>
            <configuration>
                <outputProperty>wildfly.module.classpath</outputProperty>
                <pathSeparator>:</pathSeparator>
                <excludeGroupIds>javax</excludeGroupIds>
                <excludeScope>test</excludeScope>
                <includeScope>provided</includeScope>
            </configuration>
        </execution>
    </executions>
</plugin>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-dependency-plugin</artifactId>

<version>${version.plugin.dependency}</version>

<phase>generate-sources</phase>

<goals>

<goal>build-classpath</goal>

</goals>

<outputProperty>wildfly.module.classpath</outputProperty>

<excludeGroupIds>javax</excludeGroupIds>

<includeScope>provided</includeScope>

</configuration>

</execution>

</executions>

</plugin>

This is going to pick all dependencies (including transitive), exclude javax (since they are already present in Wildfly) and exclude test scope dependencies. We need the following dependencies for Spring Batch:

<!-- Needed for Wildfly module -->
<dependency>
    <groupId>org.springframework.batch</groupId>
    <artifactId>spring-batch-core</artifactId>
    <version>3.0.0.RELEASE</version>
    <scope>provided</scope>
</dependency>

<dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring-jdbc</artifactId>
    <version>4.0.5.RELEASE</version>
    <scope>provided</scope>
</dependency>

<dependency>
    <groupId>commons-dbcp</groupId>
    <artifactId>commons-dbcp</artifactId>
    <version>1.4</version>
    <scope>provided</scope>
</dependency>

<dependency>
    <groupId>org.hsqldb</groupId>
    <artifactId>hsqldb</artifactId>
    <version>2.3.2</version>
    <scope>provided</scope>
</dependency>

<groupId>org.springframework.batch</groupId>

<artifactId>spring-batch-core</artifactId>

<version>3.0.0.RELEASE</version>

<scope>provided</scope>

</dependency>

<groupId>org.springframework</groupId>

<artifactId>spring-jdbc</artifactId>

<version>4.0.5.RELEASE</version>

<scope>provided</scope>

</dependency>

<groupId>commons-dbcp</groupId>

<artifactId>commons-dbcp</artifactId>

<scope>provided</scope>

</dependency>

<groupId>org.hsqldb</groupId>

<artifactId>hsqldb</artifactId>

<scope>provided</scope>

</dependency>

Now, we need to replace the property in the file. Let’s use Maven Resources plugin:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-resources-plugin</artifactId>
    <version>${version.plugin.resources}</version>
    <executions>
        <execution>
            <id>copy-resources</id>
            <phase>process-resources</phase>
            <goals>
                <goal>copy-resources</goal>
            </goals>
            <configuration>
                <outputDirectory>${basedir}/target/scripts</outputDirectory>
                <resources>
                    <resource>
                        <directory>src/main/resources/scripts</directory>
                        <filtering>true</filtering>
                    </resource>
                </resources>
            </configuration>
        </execution>
    </executions>
</plugin>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-resources-plugin</artifactId>

<version>${version.plugin.resources}</version>

<id>copy-resources</id>

<phase>process-resources</phase>

<goals>

<goal>copy-resources</goal>

</goals>

<outputDirectory>${basedir}/target/scripts</outputDirectory>

<directory>src/main/resources/scripts</directory>

</resource>

</resources>

</configuration>

</execution>

</executions>

</plugin>

This will filter the configured files and replace the property wildfly.module.classpath with the value we generated previously. This is a classpath pointing to the dependencies in your local Maven repository. Now with Wildfly Maven Plugin we can execute this script (you need to have Wildfly running):

<plugin>
    <groupId>org.wildfly.plugins</groupId>
    <artifactId>wildfly-maven-plugin</artifactId>
    <version>${version.plugin.wildfly}</version>
    <configuration>
        <skip>false</skip>
        <executeCommands>
            <batch>false</batch>
            <scripts>
                <!--suppress MavenModelInspection -->
                <script>target/scripts/${cli.file}</script>
            </scripts>
        </executeCommands>
    </configuration>
</plugin>

<groupId>org.wildfly.plugins</groupId>

<artifactId>wildfly-maven-plugin</artifactId>

<version>${version.plugin.wildfly}</version>

<skip>false</skip>

<batch>false</batch>

</scripts>

</executeCommands>

</configuration>

</plugin>

And these profiles:

<profiles>
    <profile>
        <id>install-spring-batch</id>
        <properties>
            <cli.file>wildfly-add-spring-batch.cli</cli.file>
        </properties>
    </profile>

    <profile>
        <id>remove-spring-batch</id>
        <properties>
            <cli.file>wildfly-remove-spring-batch.cli</cli.file>
        </properties>
    </profile>
</profiles>

<id>install-spring-batch</id>

<cli.file>wildfly-add-spring-batch.cli</cli.file>

</properties>

</profile>

<id>remove-spring-batch</id>

<cli.file>wildfly-remove-spring-batch.cli</cli.file>

</properties>

</profile>

</profiles>

(For the full pom.xml contents, check here)

We can add the module by executing:
mvn process-resources wildfly:execute-commands -P install-spring-batch.

Or remove the module by executing:
mvn wildfly:execute-commands -P remove-spring-batch.

This strategy works for any module that you want to create into Wildfly. Think about adding a JDBC driver. You usually use a module to add it into the server, but all the documentation I’ve found about this is always a manual process. This works great for CI builds, so you can have everything you need to setup your environment.

Use Spring-Batch

Ok, I have my module there, but how can I instruct Wildfly to use it instead of jBeret? We need to add a the following file in META-INF folder of our application:

jboss-deployment-structure.xml

<?xml version="1.0" encoding="UTF-8"?>
<jboss-deployment-structure>
    <deployment>
        <exclusions>
            <module name="org.wildfly.jberet"/>
            <module name="org.jberet.jberet-core"/>
        </exclusions>

        <dependencies>
            <module name="org.springframework.batch" services="import" meta-inf="import"/>
        </dependencies>
    </deployment>
</jboss-deployment-structure>

<?xml version="1.0" encoding="UTF-8"?>

<jboss-deployment-structure>

</exclusions>

</dependencies>

</deployment>

</jboss-deployment-structure>

Since the JSR-352 uses a Service Loader to load the implementation, the only possible outcome would be to load the service specified in org.springframework.batch module. Your batch code will now run with the Spring Batch implementation.

Testing

The github repository code, has Arquillian sample tests that demonstrate the behaviour. Check the Resources section below.

Resources

You can clone a full working copy from my github repository. You can find instructions there to deploy it.

Wildfly – Spring Batch

Since I may modify the code in the future, you can download the original source of this post from the release 1.0. In alternative, clone the repo, and checkout the tag from release 1.0 with the following command: git checkout 1.0.

Future

I’ve still need to apply this to the Java EE Samples. It’s on my TODO list.

Comments ( 17 )

ReplyAbhijith August 6, 2014
Awesome. Just what I was looking for.

ReplyRoberto Cortez August 7, 2014
Thank you 🙂

ReplyAnu October 7, 2015
I have one question, if a project is already using spring web-mvc of version X and adding this spring-batch of version Y ( version X > Y => X has new classes not available in Y ) from wildfly module, then the project deployment can fail as it tries to load both spring versions in same deployment class loader. Do you have any idea to fix this ?

ReplyRoberto Cortez October 18, 2015
Hi Anu,
Thank you for your question.
The answer may vary, depending on your setup. I’m assuming that you’re using spring mvc for some web application to call a spring batch job. You could place the spring batch job into another WAR/EAR and expose an EJB to perform the call. WARs and EARs have separate classloaders, so it should fix your problem.
Cheers,

Replysid anyone January 6, 2016
Thanks for the good example. This runs in some minutes at the wildfly server. But I have a problem. We build a very big EAR with some JARs and WARs.
With this example I have created a JMX MBean in a separate WAR with an operation to start the job. The jobs are defined in this WAR. Both WARs are packed in an EAR an deployed on Wildfly.
If I execute the MBean operation then an exeption is throw because org.jberet is used:
Problem invoking startJob: javax.management.RuntimeErrorException: java.util.ServiceConfigurationError: javax.batch.operations.JobOperator: Provider org.jberet.operations.JobOperatorImpl could not be instantiated
I think that all files (beans.xml, jboss-deployment-structure.xml, batch.xml, batchlet-job.xml) at the correct place.
com.cortez.wildfly.batch.ear
META-INF
jboss-deployment-structure.xml
batch.xml
batch-jobs\batchlet-job.xml
MANIFEST.MF
WEB-INF
beans.xml
com.cortez.wildfly.batch.jmx.war
META-INF
MANIFEST.MF
WEB-INF
web.xml
classes
com\cortez\wildfly\batch\batchlet\ApplicationMBeanLifeCycleListener.class
com\cortez\wildfly\batch\batchlet\MyBatchletService.class
com\cortez\wildfly\batch\batchlet\MyBatchletServiceMBean.class
com.cortez.wildfly.batch.war.war
META-INF
MANIFEST.MF
WEB-INF
classes
com\cortez\wildfly\batch\batchlet\MyBatchlet.class
Do you have any idea to fix this? Thanks.

ReplyRoberto Cortez January 7, 2016
Hi sid,
Usually, war files are class loader isolated, so they don’t “see” each other. That would be my first guest, but it’s hard to figure out, just with this information.
Do you have a chance to fork my project and push your changes so I can have a look?
Cheers,
Roberto

Replysid anyone January 8, 2016
Hello Roberto, I wrote you a mail with my example. Thanks for your reply.

ReplyRoberto Cortez January 8, 2016
Hi Sid,
Thank you. It’s a fun problem. You should have a solution in your email. I guess I would have to write a blog post about it 🙂
Cheers,
Roberto

Replysid anyone January 18, 2016
I think I’ve found a solution. In the EAR I must include the file src/main/application/META-INF/jboss-deployment-structure.xml with
<?xml version=”1.0″ encoding=”UTF-8″?>
<jboss-deployment-structure>
<sub-deployment name=”…”>
  <exclusions>
   <module name=”org.wildfly.jberet” />
   <module name=”org.jberet.jberet-core” />
  </exclusions>
  <dependencies>
   <module name=”org.springframework.batch” services=”import” meta-inf=”import” />
  </dependencies>
</sub-deployment>
</jboss-deployment-structure>
Now the BatchRuntime with org.springframework.batch… is used.

ReplyRoberto Cortez January 21, 2016
Hi Sid,
Great! Thanks for posting the solution 🙂
Cheers,
Roberto

Replyjamezp May 16, 2016
I’d just like to add that in new versions of WildFly either the batch-jberet subsystem needs to be removed or excluded from deployments.

Note in WildFly 10 the subsystem was renamed from batch to batch-jberet. Versions earlier than 10 will need to exclude the batch subsystem.

Replyjamezp May 16, 2016
Sorry the XML didn’t seem to come through. I’ll try one more time
<jboss-deployment-structure> <deployment> <exclude-subsystems> <subsystem name="batch-jberet"/> </exclude-subsystems> </deployment> </jboss-deployment-structure>

ReplyRoberto Cortez May 23, 2016
Hi Mauri,
Thank you for your note. In fact, I haven’t tried the project with Wilfly 10, so I’m glad you did and submitted a patch.
Cheers,
Roberto

ReplyRaja April 14, 2017
Hi
When I running the test with wildfly 10.1.0.Final I am getting the below exception.
23:10:24,621 WARN [org.jboss.modules] (Weld Thread Pool — 1) Failed to define class com.cortez.wildfly.batch.batchlet.MyBatchlet in Module “deployment.6bdf5a94-4922-4034-b522-8953db21ccfb.war:main” from Service Module Loader: java.lang.NoClassDefFoundError: Failed to link com/cortez/wildfly/batch/batchlet/MyBatchlet (Module “deployment.6bdf5a94-4922-4034-b522-8953db21ccfb.war:main” from Service Module Loader): javax/batch/api/AbstractBatchlet
Any suggestion will be appreciated.
Thanks
Raja

ReplyRoberto Cortez December 1, 2017
Hum… I did this with a previous Wildfly version. It might not be working with the most recent ones. I’ll have to check.

ReplyRaja April 14, 2017
I saw this log also in the console
23:10:24,637 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool — 1) WELD-000119: Not generating any bean definitions from com.cortez.wildfly.batch.batchlet.MyBatchlet because of underlying class loading error: Type Failed to link com.cortez.wildfly.batch.batchlet.MyBatchlet (Module “deployment.6bdf5a94-4922-4034-b522-8953db21ccfb.war:main” from Service Module Loader): javax.batch.api.AbstractBatchlet not found. If this is unexpected, enable DEBUG logging to see the full error.

ReplyRoberto Cortez December 1, 2017
It seems that the Batch API is not loaded with the app. There might be some difference between Wildfly versions. I’ll try to check it out.
Thanks.

Roberto Cortez Java Blog

Spring Batch as Wildfly Module

What’s the point?

Motivation

Code

Adding Wildfly Module with Maven

Use Spring-Batch

Testing

Resources

Future

Comments ( 17 )

Leave a Reply to Roberto Cortez Cancel reply