Roberto Cortez

Published: 76 articles

Java EE 7 Batch Processing and World of Warcraft – Part 2

posted by Roberto Cortez on January 8, 2015

Today, I bring you the second part to my previous post about Java EE 7 Batch Processing and World of Warcraft – Part 1. In this post, we are going to see how to aggregate and extract metrics from the data that we obtained in Part 1.

Recap

The batch purpose is to download the World of Warcraft Auction House’s data, process the auctions and extract metrics. These metrics are going to build a history of the Auctions Items price evolution through time. In Part 1, we already downloaded and inserted the data into a database.

The Application

Process Job

After adding the raw data into the database, we are going to add another step with a Chunk style processing. In the chunk we’re are going to read the aggregated data, and then insert it into another table in the database for easy access. This is done in the process-job.xml:

<step id="importStatistics">
    <chunk item-count="100">
        <reader ref="processedAuctionsReader"/>
        <processor ref="processedAuctionsProcessor"/>
        <writer ref="processedAuctionsWriter"/>
    </chunk>
</step>

</chunk>

</step>

A Chunk reads the data one item at a time, and creates chunks that will be written out, within a transaction. One item is read in from an ItemReader, handed to an ItemProcessor, and aggregated. Once the number of items read equals the commit interval, the entire chunk is written out via the ItemWriter, and then the transaction is committed.

ProcessedAuctionsReader

In the reader, we are going to select and aggregate metrics using database functions.

@Named
public class ProcessedAuctionsReader extends AbstractAuctionFileProcess implements ItemReader {
    @Resource(name = "java:comp/DefaultDataSource")
    protected DataSource dataSource;

    private PreparedStatement preparedStatement;
    private ResultSet resultSet;

    @Override
    public void open(Serializable checkpoint) throws Exception {
        Connection connection = dataSource.getConnection();

        preparedStatement = connection.prepareStatement(
                        "SELECT" +
                        "   itemid as itemId," +
                        "   sum(quantity)," +
                        "   sum(bid)," +
                        "   sum(buyout)," +
                        "   min(bid / quantity)," +
                        "   min(buyout / quantity)," +
                        "   max(bid / quantity)," +
                        "   max(buyout / quantity)" +
                        " FROM auction" +
                        " WHERE auctionfile_id = " +
                        getContext().getFileToProcess().getId() +
                        " GROUP BY itemid" +
                        " ORDER BY 1",
                ResultSet.TYPE_FORWARD_ONLY,
                ResultSet.CONCUR_READ_ONLY,
                ResultSet.HOLD_CURSORS_OVER_COMMIT
        );

        // Weird bug here. Check https://java.net/bugzilla/show_bug.cgi?id=5315
        //preparedStatement.setLong(1, getContext().getFileToProcess().getId());

        resultSet = preparedStatement.executeQuery();
    }

    @Override
    public void close() throws Exception {
        DbUtils.closeQuietly(resultSet);
        DbUtils.closeQuietly(preparedStatement);
    }

    @Override
    public Object readItem() throws Exception {
        return resultSet.next() ? resultSet : null;
    }

    @Override
    public Serializable checkpointInfo() throws Exception {
        return null;
    }
}

@Named

public class ProcessedAuctionsReader extends AbstractAuctionFileProcess implements ItemReader {

@Resource(name = "java:comp/DefaultDataSource")

protected DataSource dataSource;

private PreparedStatement preparedStatement;

private ResultSet resultSet;

@Override

public void open(Serializable checkpoint) throws Exception {

Connection connection = dataSource.getConnection();

preparedStatement = connection.prepareStatement(

"SELECT" +

" itemid as itemId," +

" sum(quantity)," +

" sum(bid)," +

" sum(buyout)," +

" min(bid / quantity)," +

" min(buyout / quantity)," +

" max(bid / quantity)," +

" max(buyout / quantity)" +

" FROM auction" +

" WHERE auctionfile_id = " +

getContext().getFileToProcess().getId() +

" GROUP BY itemid" +

" ORDER BY 1",

ResultSet.TYPE_FORWARD_ONLY,

ResultSet.CONCUR_READ_ONLY,

ResultSet.HOLD_CURSORS_OVER_COMMIT

);

// Weird bug here. Check https://java.net/bugzilla/show_bug.cgi?id=5315

//preparedStatement.setLong(1, getContext().getFileToProcess().getId());

resultSet = preparedStatement.executeQuery();

}

@Override

public void close() throws Exception {

DbUtils.closeQuietly(resultSet);

DbUtils.closeQuietly(preparedStatement);

}

@Override

public Object readItem() throws Exception {

return resultSet.next() ? resultSet : null;

}

@Override

public Serializable checkpointInfo() throws Exception {

return null;

}

For this example, we get the best performance results by using plain JDBC with a simple scrollable result set. In this way, only one query is executed and results are pulled as needed in readItem. You might want to explore other alternatives.

Plain JPA doesn’t have a scrollable result set in the standards, so you need to paginate the results. This will lead to multiple queries which will slow down the reading. Another option is to use the new Java 8 Streams API to perform the aggregation operations. The operations are quick, but you need to select the entire dataset from the database into the streams. Ultimately, this will kill your performance.

I did try both approaches and got the best results by using the database aggregation capabilities. I’m not saying that this is always the best option, but in this particular case it was the best option.

During the implementation, I’ve also found a bug in Batch. You can check it here. An exception is thrown when setting parameters in the PreparedStatement. The workaround was to inject the parameters directly into the query SQL. Ugly, I know…

ProcessedAuctionsProcessor

In the processor, let’s store all the aggregated values in a holder object to store in the database.

@Named
public class ProcessedAuctionsProcessor extends AbstractAuctionFileProcess implements ItemProcessor {
    @Override
    @SuppressWarnings("unchecked")
    public Object processItem(Object item) throws Exception {
        ResultSet resultSet = (ResultSet) item;

        AuctionItemStatistics auctionItemStatistics = new AuctionItemStatistics();
        auctionItemStatistics.setItemId(resultSet.getInt(1));
        auctionItemStatistics.setQuantity(resultSet.getLong(2));
        auctionItemStatistics.setBid(resultSet.getLong(3));
        auctionItemStatistics.setBuyout(resultSet.getLong(4));
        auctionItemStatistics.setMinBid(resultSet.getLong(5));
        auctionItemStatistics.setMinBuyout(resultSet.getLong(6));
        auctionItemStatistics.setMaxBid(resultSet.getLong(7));
        auctionItemStatistics.setMaxBuyout(resultSet.getLong(8));

        auctionItemStatistics.setTimestamp(getContext().getFileToProcess().getLastModified());

        auctionItemStatistics.setAvgBid(
                (double) (auctionItemStatistics.getBid() / auctionItemStatistics.getQuantity()));
        auctionItemStatistics.setAvgBuyout(
                (double) (auctionItemStatistics.getBuyout() / auctionItemStatistics.getQuantity()));

        auctionItemStatistics.setRealm(getContext().getRealm());

        return auctionItemStatistics;
    }
}

@Named

public class ProcessedAuctionsProcessor extends AbstractAuctionFileProcess implements ItemProcessor {

@Override

@SuppressWarnings("unchecked")

public Object processItem(Object item) throws Exception {

ResultSet resultSet = (ResultSet) item;

AuctionItemStatistics auctionItemStatistics = new AuctionItemStatistics();

auctionItemStatistics.setItemId(resultSet.getInt(1));

auctionItemStatistics.setQuantity(resultSet.getLong(2));

auctionItemStatistics.setBid(resultSet.getLong(3));

auctionItemStatistics.setBuyout(resultSet.getLong(4));

auctionItemStatistics.setMinBid(resultSet.getLong(5));

auctionItemStatistics.setMinBuyout(resultSet.getLong(6));

auctionItemStatistics.setMaxBid(resultSet.getLong(7));

auctionItemStatistics.setMaxBuyout(resultSet.getLong(8));

auctionItemStatistics.setTimestamp(getContext().getFileToProcess().getLastModified());

auctionItemStatistics.setAvgBid(

(double) (auctionItemStatistics.getBid() / auctionItemStatistics.getQuantity()));

auctionItemStatistics.setAvgBuyout(

(double) (auctionItemStatistics.getBuyout() / auctionItemStatistics.getQuantity()));

auctionItemStatistics.setRealm(getContext().getRealm());

return auctionItemStatistics;

}

Since the metrics record an exact snapshot of the data in time, the calculation only needs to be done once. That’s why we are saving the aggregated metrics. They are never going to change and we can easily check the history.

If you know that your source data is immutable and you need to perform operations on it, I recommend that you persist the result somewhere. This is going to save you time. Of course, you need to balance if this data is going to be accessed many times in the future. If not, maybe you don’t need to go through the trouble of persisting the data.

ProcessedAuctionsWriter

Finally we just need to write the data down to a database:

@Named
public class ProcessedAuctionsWriter extends AbstractItemWriter {
    @PersistenceContext
    protected EntityManager em;

    @Override
    @SuppressWarnings("unchecked")
    public void writeItems(List items) throws Exception {
        List<AuctionItemStatistics> statistis = (List<AuctionItemStatistics>) items;
        statistis.forEach(em::persist);
    }
}

@Named

public class ProcessedAuctionsWriter extends AbstractItemWriter {

@PersistenceContext

protected EntityManager em;

@Override

@SuppressWarnings("unchecked")

public void writeItems(List items) throws Exception {

List<AuctionItemStatistics> statistis = (List<AuctionItemStatistics>) items;

statistis.forEach(em::persist);

}

Metrics

Now, to do something useful with the data we are going to expose a REST endpoint to perform queries on the calculated metrics. Here is how:

    @Override @GET
    @Path("items")
    public List<AuctionItemStatistics> findAuctionItemStatisticsByRealmAndItem(@QueryParam("realmId") Long realmId,
                                                                               @QueryParam("itemId") Integer itemId) {

        Realm realm = (Realm) em.createNamedQuery("Realm.findRealmsWithConnectionsById")
                                .setParameter("id", realmId)
                                .getSingleResult();

        // Workaround for https://bugs.eclipse.org/bugs/show_bug.cgi?id=433075 if using EclipseLink
        List<Realm> connectedRealms = new ArrayList<>();
        connectedRealms.addAll(realm.getConnectedRealms());
        List<Long> ids = connectedRealms.stream().map(Realm::getId).collect(Collectors.toList());
        ids.add(realmId);

        return em.createNamedQuery("AuctionItemStatistics.findByRealmsAndItem")
                 .setParameter("realmIds", ids)
                 .setParameter("itemId", itemId)
                 .getResultList();
    }

@Override @GET

@Path("items")

public List<AuctionItemStatistics> findAuctionItemStatisticsByRealmAndItem(@QueryParam("realmId") Long realmId,

@QueryParam("itemId") Integer itemId) {

Realm realm = (Realm) em.createNamedQuery("Realm.findRealmsWithConnectionsById")

.setParameter("id", realmId)

.getSingleResult();

// Workaround for https://bugs.eclipse.org/bugs/show_bug.cgi?id=433075 if using EclipseLink

List<Realm> connectedRealms = new ArrayList<>();

connectedRealms.addAll(realm.getConnectedRealms());

List<Long> ids = connectedRealms.stream().map(Realm::getId).collect(Collectors.toList());

ids.add(realmId);

return em.createNamedQuery("AuctionItemStatistics.findByRealmsAndItem")

.setParameter("realmIds", ids)

.setParameter("itemId", itemId)

.getResultList();

}

If you remember a few details of Part 1 post, World of Warcraft servers are called Realms. These realms can be linked with each other and share the same Auction House. To that end, we also have information on how the realms connect with each other. This is important, because we can search for an Auction Item in all the realms that are connected. The rest of the logic is just simple queries to get the data out.

During development, I’ve also found a bug with Eclipse Link (if you run in Glassfish) and Java 8. Apparently the underlying Collection returned by Eclipse Link has the element count set to 0. This doesn’t work well with Streams if you try to inline the query call plus a Stream operation. The Stream will think that it’s empty and no results are returned. You can read a little more about this here.

Interface

I’ve also developed a small interface using Angular and Google Charts to display the metrics. Have a look:

In here, I’m searching in the Realm named “Aggra (Português)” and the Auction Item id 72092 which corresponds to Ghost Iron Ore. As you can see, we can check the quantity for sale, bid and buyout values and price fluctuation through time. Neat? I may write another post about building the Web Interface in the future.

Resources

You can clone a full working copy from my github repository and deploy it to Wildfly or Glassfish. You can find instructions there to deploy it:

World of Warcraft Auctions

Check also the Java EE samples project, with a lot of batch examples, fully documented.

SSL / TLS REST Server – Client with Spring and TomEE

posted by Roberto Cortez on December 23, 2014

20 Comments

tags: REST, security, spring, tomee

When building a system, developers usually disregard the security aspects. Security has been always something very important to worry about, but it’s attracting even higher concerns than before. Just this year we had a few cases like the Heartbleed Bug or the CelebrityGate scandal. This has nothing to do with the post, but are just examples that security really matters and we should be aware of it.

With the increasing popularity of REST services it makes sense that these need to be secured in some way. A couple of weeks ago, I had to integrate my client with a REST service behind https. I have never done it before and that’s the reason for this post. I have to confess that I’m no security expert myself, so please correct me if I write anything stupid.

The Setup

For this example I have used the following setup:

TomEE (or Tomcat) with SSL Configuration
Spring
Apache HTTP Components

I’m not going into many details about SSL and TSL, so please check here for additional content. Note that TLS is the new name for SSL evolution. Sometimes there is confusion between the two and people often say SSL, but use the newest version of TSL. Keep that in mind.

Don’t forget to follow the instructions on the following page to setup SSL for Tomcat: SSL Configuration HOW-TO. This is needed for the server to present the client with a set of credentials, a Certificate, to secure the connection between server and client.

The Code

Service

Let’s create a simple Spring REST Service:

RestService.java

@Controller
@RequestMapping("/")
public class RestService {
    @RequestMapping(method = RequestMethod.GET)
    @ResponseBody
    public String get() {
        return "Called the get Rest Service";
    }
}

@Controller

@RequestMapping("/")

public class RestService {

@RequestMapping(method = RequestMethod.GET)

@ResponseBody

public String get() {

return "Called the get Rest Service";

}

And we also need some wiring for this to work:

RestConfig.java

@Configuration
@EnableWebMvc
@ComponentScan(basePackages = "com.radcortez.rest.ssl")
public class RestConfig {}

@Configuration

@EnableWebMvc

@ComponentScan(basePackages = "com.radcortez.rest.ssl")

public class RestConfig {}

web.xml

<?xml version="1.0" encoding="UTF-8"?>
<web-app
    version="3.1"
    xmlns="http://xmlns.jcp.org/xml/ns/javaee"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd">

    <servlet>
        <servlet-name>rest</servlet-name>
        <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
        <init-param>
            <param-name>contextClass</param-name>
            <param-value>org.springframework.web.context.support.AnnotationConfigWebApplicationContext</param-value>
        </init-param>
        <init-param>
            <param-name>contextConfigLocation</param-name>
            <param-value>com.radcortez.rest.ssl</param-value>
        </init-param>
        <load-on-startup>1</load-on-startup>
    </servlet>

    <servlet-mapping>
        <servlet-name>rest</servlet-name>
        <url-pattern>/</url-pattern>
    </servlet-mapping>

    <security-constraint>
        <web-resource-collection>
            <web-resource-name>Rest Application</web-resource-name>
            <url-pattern>/*</url-pattern>
        </web-resource-collection>
        <user-data-constraint>
            <!-- Needed for our application to respond to https requests -->
            <transport-guarantee>CONFIDENTIAL</transport-guarantee>
        </user-data-constraint>
    </security-constraint>
</web-app>

<?xml version="1.0" encoding="UTF-8"?>

<web-app

version="3.1"

xmlns="http://xmlns.jcp.org/xml/ns/javaee"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd">

<servlet-name>rest</servlet-name>

<servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>

<init-param>

<param-name>contextClass</param-name>

<param-value>org.springframework.web.context.support.AnnotationConfigWebApplicationContext</param-value>

</init-param>

<init-param>

<param-name>contextConfigLocation</param-name>

<param-value>com.radcortez.rest.ssl</param-value>

</init-param>

<load-on-startup>1</load-on-startup>

</servlet>

<servlet-mapping>

<servlet-name>rest</servlet-name>

<url-pattern>/</url-pattern>

</servlet-mapping>

<security-constraint>

<web-resource-collection>

<web-resource-name>Rest Application</web-resource-name>

<url-pattern>/*</url-pattern>

</web-resource-collection>

<user-data-constraint>

<transport-guarantee>CONFIDENTIAL</transport-guarantee>

</user-data-constraint>

</security-constraint>

</web-app>

Please, note the elements security-constraint, user-data-constraint and <transport-guarantee>CONFIDENTIAL</transport-guarantee>. These are needed to specify that the application requires a secure connection. Check Securing Web Applications for Java Applications.

Running the Service

Just deploy the application on the TomEE Server using your favourite IDE environment and access https://localhost:8443/. You should get the following (you might need to accept the server certificate first):

Note that the browser protocol is https and the port is 8443 (assuming that you kept the default settings in SSL Configuration HOW-TO.

Client

Now, if you try to call this REST service with a Java client, most likely you are going to get the following message and Exception (or similar):

Message: I/O error on GET request for “https://localhost:8443/”:sun.security.validator.ValidatorException:

Exception: Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

This happens because the running JDK does not have a valid certificate for your server. You can import it, and get rid of the problem, but let’s do something more interesting. We are going to programatically supply a trusted keystore with our server certificate.

This is especially useful if:

you are running your code into multiple environments
you don’t have to manually import the certificate into the JDK every time
if you upgrade the JDK you have to remember about the certificates
for some odd reason you don’t have access to the JDK itself to import the certificate

Let’s write some code:

RestClientConfig.java

@Configuration
@PropertySource("classpath:config.properties")
public class RestClientConfig {
    @Bean
    public RestOperations restOperations(ClientHttpRequestFactory clientHttpRequestFactory) throws Exception {
        return new RestTemplate(clientHttpRequestFactory);
    }

    @Bean
    public ClientHttpRequestFactory clientHttpRequestFactory(HttpClient httpClient) {
        return new HttpComponentsClientHttpRequestFactory(httpClient);
    }

    @Bean
    public HttpClient httpClient(@Value("${keystore.file}") String file,
                                 @Value("${keystore.pass}") String password) throws Exception {
        KeyStore trustStore = KeyStore.getInstance(KeyStore.getDefaultType());
        FileInputStream instream = new FileInputStream(new File(file));
        try {
            trustStore.load(instream, password.toCharArray());
        } finally {
            instream.close();
        }

        SSLContext sslcontext =
                SSLContexts.custom().loadTrustMaterial(trustStore, new TrustSelfSignedStrategy()).build();
        SSLConnectionSocketFactory sslsf =
                new SSLConnectionSocketFactory(sslcontext, new String[]{"TLSv1.2"}, null,
                                               BROWSER_COMPATIBLE_HOSTNAME_VERIFIER);
        return HttpClients.custom().setSSLSocketFactory(sslsf).build();
    }

    @Bean
    public static PropertySourcesPlaceholderConfigurer propertySourcesPlaceholderConfigurer() {
        return new PropertySourcesPlaceholderConfigurer();
    }
}

@Configuration

@PropertySource("classpath:config.properties")

public class RestClientConfig {

@Bean

public RestOperations restOperations(ClientHttpRequestFactory clientHttpRequestFactory) throws Exception {

return new RestTemplate(clientHttpRequestFactory);

}

@Bean

public ClientHttpRequestFactory clientHttpRequestFactory(HttpClient httpClient) {

return new HttpComponentsClientHttpRequestFactory(httpClient);

}

@Bean

public HttpClient httpClient(@Value("${keystore.file}") String file,

@Value("${keystore.pass}") String password) throws Exception {

KeyStore trustStore = KeyStore.getInstance(KeyStore.getDefaultType());

FileInputStream instream = new FileInputStream(new File(file));

try {

trustStore.load(instream, password.toCharArray());

} finally {

instream.close();

}

SSLContext sslcontext =

SSLContexts.custom().loadTrustMaterial(trustStore, new TrustSelfSignedStrategy()).build();

SSLConnectionSocketFactory sslsf =

new SSLConnectionSocketFactory(sslcontext, new String[]{"TLSv1.2"}, null,

BROWSER_COMPATIBLE_HOSTNAME_VERIFIER);

return HttpClients.custom().setSSLSocketFactory(sslsf).build();

}

@Bean

public static PropertySourcesPlaceholderConfigurer propertySourcesPlaceholderConfigurer() {

return new PropertySourcesPlaceholderConfigurer();

}

Here we use Spring RestOperations interface which specified a basic set of RESTful operations. Next we use Apache HTTP Components SSLConnectionSocketFactory which gives us the ability to validate the identity of the server against a list of trusted certificates. The certificate is loaded from the same file used on the server by KeyStore.

RestServiceClientIT.java

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(classes = RestClientConfig.class)
public class RestServiceClientIT {
    @Autowired
    private RestOperations rest;

    @Test
    public void testRestRequest() throws Exception {
        ResponseEntity<String> response = rest.getForEntity("https://localhost:8443/", String.class);
        System.out.println("response = " + response);
        System.out.println("response.getBody() = " + response.getBody());
    }
}

@RunWith(SpringJUnit4ClassRunner.class)

@ContextConfiguration(classes = RestClientConfig.class)

public class RestServiceClientIT {

@Autowired

private RestOperations rest;

@Test

public void testRestRequest() throws Exception {

ResponseEntity<String> response = rest.getForEntity("https://localhost:8443/", String.class);

System.out.println("response = " + response);

System.out.println("response.getBody() = " + response.getBody());

}

A simple test class. We also need a properties file with the keystore file location and password:

config.properties

keystore.file=${user.home}/.keystore
keystore.pass=changeit

1 2	keystore.file=${user.home}/.keystore keystore.pass=changeit

This should work fine if you used all the defaults.

Running the Test

If you now run the test which invokes the REST service within a Java client, you should get the following output:

Response: <200 OK,Called the get Rest Service,{Server=[Apache-Coyote/1.1], Cache-Control=[private], Expires=[Thu, 01 Jan 1970 01:00:00 WET], Content-Type=[text/plain;charset=ISO-8859-1], Content-Length=[27], Date=[Tue, 23 Dec 2014 01:29:20 GMT]}>

Body: Called the get Rest Service

Conclusion

That’s it! You can now call your REST service with your client in a secured manner. If you prefer to add the certificate to the JDK keystore, please check this post.

Stay tuned for an equivalent for Java EE JAX-RS equivalent.

Resources

You can clone a full working copy from my github repository: REST SSL.

One year of Blog

posted by Roberto Cortez on December 15, 2014

10 Comments

tags: blog

Hi everyone! Exactly one year ago, I’ve launched this blog. Looking into my first post, I’ve written the following:

I have created this blog to share any Java related topics to anyone that may care. For me, it would be useful to improve my English and keep a record of stuff I work and do. I usually forget a lot of stuff!

After one year, I think this is still true. I would like to think that my English improved because of the writing. This blog has now been actually one of my most valuable resources. Even yesterday I was looking into one of my older posts because I couldn’t remember a few details about it.

But more important are the Readers! And that is you! The feedback has been overwhelming positive, and I could not Thank You enough for taking your time to read my stuff.

A few numbers:

39 posts
114 comments
83819 page views
166 visiting countries

Most popular posts:

Angular is really popular. I guess I have to write a few more things about it!

My favourite posts:

Final Words

That’s it! I hope that I can write more awesome content for the next year!

Seventh Coimbra JUG Meeting – Forge and First Anniversary

posted by Roberto Cortez on December 12, 2014

0 Comments

tags: coimbrajug, forge

Last Thursday, 11 December 2014, the seventh meeting of Coimbra JUG was held on the Department of Informatics Engineering of the University of Coimbra, in Portugal. This was a special event, since we had the pleasure to have Koen Aers, the first international speaker to visit our JUG, talk a little bit about a very promising tool: JBoss Forge. This meeting was also our First Anniversary Celebration. That’s right, Coimbra JUG was created last year in December. For this meeting we had around 30 people and a few newcomers.

A few people in the audience had already heard about JBoss Forge, but no one was actively using it. Koen demonstrated the tool capabilities by creating a Java EE application, with a few entities and scaffolded a JSF interface in a short amount of time. Next, added some REST services and showed us a way to generate JPA entities with an existing database schema. Time for a break!

Thanks to some of our sponsors we had a few snacks and beer. Of course, we also had a cake to celebrate our first anniversary.

Back to JBoss Forge, Koen was now showing us how to develop plugins. The attendees seemed very curious and interested to learn about it and the questions were great. A lot of interactions and discussions were generated during the session. I might write something about JBoss Forge in the future.

Here are the materials for the session:

Enjoy!

Acknowledgements

I would like to thanks all the sponsors, that in one way or another contributed to our JUG during our first year:

Critical Software for offering the snacks
Cervejaria Praxis for offering the beer
JetBrains for IntelliJ licenses
O’Reilly for books
Javaland Conference for event tickets
Devoxx Conference for event tickets
JCP for merchandise

And of course Koen Aers for taking is time to visit us here in Coimbra. Thank you!

About Coimbra JUG

First Meeting in 19/12/2013
A meeting every 2 months
150 Members
280 participants in all Meetings
Meetup – http://www.meetup.com/Coimbra-JUG/
Mailing List – coimbra-jug-list@meetup.com
Youtube – http://www.youtube.com/user/coimbrajug
Twitter – @coimbrajug

Annotations, Annotations Everywhere

posted by Roberto Cortez on December 4, 2014

17 Comments

tags: java

Annotations became available with Java 1.5 in 2004, ten years ago. It’s hard to imagine our code without this feature. In fact, annotations were first introduced to relieve developers from the pain of writing tedious boilerplate code and make the code more readable. Think about J2EE 1.4 (no annotations available) and Java EE 5. Annotation adoption considerably simplified the development of a Java EE application by getting rid of all the configuration XML. Even today, more annotations are being added to the newest version of Java EE. The idea is to unburden the developer and increase productivity. Other technologies and frameworks use them extensively as well.

Annotations Everywhere

Let’s see an example on how annotations have simplified our code (from my post about JPA Entity Graphs):

@Entity
@Table(name = "MOVIE_ENTITY_GRAPH")
@NamedQueries({
    @NamedQuery(name = "Movie.findAll", query = "SELECT m FROM Movie m")
})
@NamedEntityGraphs({
    @NamedEntityGraph(
        name = "movieWithActors",
        attributeNodes = {
            @NamedAttributeNode("movieActors")
        }
    ),
    @NamedEntityGraph(
        name = "movieWithActorsAndAwards",
        attributeNodes = {
            @NamedAttributeNode(value = "movieActors", subgraph = "movieActorsGraph")
        },
        subgraphs = {
            @NamedSubgraph(
                    name = "movieActorsGraph",
                    attributeNodes = {
                        @NamedAttributeNode("movieActorAwards")
                    }
            )
        }
    )
})
public class Movie implements Serializable {
    @Id
    private Integer id;

    @NotNull
    @Size(max = 50)
    private String name;

    @OneToMany
    @JoinColumn(name = "ID")
    private Set<MovieActor> movieActors;

    @OneToMany(fetch = FetchType.EAGER)
    @JoinColumn(name = "ID")
    private Set<MovieDirector> movieDirectors;

    @OneToMany
    @JoinColumn(name = "ID")
    private Set<MovieAward> movieAwards;
}

@Entity

@Table(name = "MOVIE_ENTITY_GRAPH")

@NamedQueries({

@NamedQuery(name = "Movie.findAll", query = "SELECT m FROM Movie m")

})

@NamedEntityGraphs({

@NamedEntityGraph(

name = "movieWithActors",

attributeNodes = {

@NamedAttributeNode("movieActors")

}

@NamedEntityGraph(

name = "movieWithActorsAndAwards",

attributeNodes = {

@NamedAttributeNode(value = "movieActors", subgraph = "movieActorsGraph")

subgraphs = {

@NamedSubgraph(

name = "movieActorsGraph",

attributeNodes = {

@NamedAttributeNode("movieActorAwards")

}

)

}

)

})

public class Movie implements Serializable {

@Id

private Integer id;

@NotNull

@Size(max = 50)

private String name;

@OneToMany

@JoinColumn(name = "ID")

private Set<MovieActor> movieActors;

@OneToMany(fetch = FetchType.EAGER)

@JoinColumn(name = "ID")

private Set<MovieDirector> movieDirectors;

@OneToMany

@JoinColumn(name = "ID")

private Set<MovieAward> movieAwards;

}

Wait a minute! Simplified? Really? Aren’t annotations supposed to make my code more readable? This example has more annotations than actual code. To be fair, I’m not including getters and setters. Also, some of the annotated code could be better condensed, but it would make the code harder to read. Of course, this is an extreme case. Anyway, I’m happy since I won the title Annotatiomaniac of the Year. Thanks Lukas!

We rely in annotations so much that we end up abusing them. It’s funny that annotations in some cases are causing the same problems that they intended to solve.

What if?

Let’s rewrite the previous sample like this:

@MovieEntity
@FindAll
@LoadWithActors
@LoadWithActorsAndAwards
public class Movie implements Serializable {
    @Id
    private Integer id;

    @Name
    private String name;

    @MovieActors
    private Set<MovieActor> movieActors;

    @MovieDirectors
    private Set<MovieDirector> movieDirectors;

    @MovieAwards
    private Set<MovieAward> movieAwards;
}

@MovieEntity

@FindAll

@LoadWithActors

@LoadWithActorsAndAwards

public class Movie implements Serializable {

@Id

private Integer id;

@Name

private String name;

@MovieActors

private Set<MovieActor> movieActors;

@MovieDirectors

private Set<MovieDirector> movieDirectors;

@MovieAwards

private Set<MovieAward> movieAwards;

}

It sure looks more readable. But these annotations do not exist. Where do they come from?

@LoadWithActors

@NamedEntityGraph(
        name = "movieWithActors",
        attributeNodes = {
                @NamedAttributeNode("movieActors")
        }
)
public @interface LoadWithActors {}

@NamedEntityGraph(

name = "movieWithActors",

attributeNodes = {

@NamedAttributeNode("movieActors")

}

)

public @interface LoadWithActors {}

@LoadWithActorsAndAwards

@NamedEntityGraph(
        name = "movieWithActorsAndAwards",
        attributeNodes = {
                @NamedAttributeNode(value = "movieActors", subgraph = "movieActorsGraph")
        },
        subgraphs = {
                @NamedSubgraph(
                        name = "movieActorsGraph",
                        attributeNodes = {
                                @NamedAttributeNode("movieActorAwards")
                        }
                )
        }
)
public @interface LoadWithActorsAndAwards {}

@NamedEntityGraph(

name = "movieWithActorsAndAwards",

attributeNodes = {

@NamedAttributeNode(value = "movieActors", subgraph = "movieActorsGraph")

subgraphs = {

@NamedSubgraph(

name = "movieActorsGraph",

attributeNodes = {

@NamedAttributeNode("movieActorAwards")

}

)

}

)

public @interface LoadWithActorsAndAwards {}

And so on for the rest. You get the feeling. The idea would be to extract and group annotation metadata into your own custom annotations. Your annotation could then be used to represent all the annotated data in your code, making it easier to understand. It’s like Java 8 Lambdas, read like the problem statement.

Just another example:

@Named
@Stateless
@TransactionAttribute(TransactionAttributeType.SUPPORTS)
@ApplicationPath("/resources")
@Path("wowauctions")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.APPLICATION_JSON)
public class WoWBusinessBean extends Application implements WoWBusiness {}

@Named

@Stateless

@TransactionAttribute(TransactionAttributeType.SUPPORTS)

@ApplicationPath("/resources")

@Path("wowauctions")

@Consumes(MediaType.APPLICATION_JSON)

@Produces(MediaType.APPLICATION_JSON)

public class WoWBusinessBean extends Application implements WoWBusiness {}

Rewritten:

@RestStatelessBean("wowauctions")
public class WoWBusinessBean extends Application implements WoWBusiness {}

1 2	@RestStatelessBean("wowauctions") public class WoWBusinessBean extends Application implements WoWBusiness {}

@RestStatelessBean

@Named
@Stateless
@TransactionAttribute(TransactionAttributeType.SUPPORTS)
@ApplicationPath("/resources")
@Path("#{path}")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.APPLICATION_JSON)
public @interface RestStatelessBean {
    String value() default "#{path}";
}

@Named

@Stateless

@TransactionAttribute(TransactionAttributeType.SUPPORTS)

@ApplicationPath("/resources")

@Path("#{path}")

@Consumes(MediaType.APPLICATION_JSON)

@Produces(MediaType.APPLICATION_JSON)

public @interface RestStatelessBean {

String value() default "#{path}";

}

Usually, I write these annotations once to define the behaviour of my POJO and never look into them again. How cool would be if I could just reuse one annotation for all my Stateless Rest Services?

Another nice effect, is that annotation configuration metadata is not directly tied in the code. Instead it’s abstracted away into a different annotation. In this case, it would be possible to override or replace the values in compile / runtime.

Meta Annotations

I first heard about this concept by David Blevins. He wrote a very good post about these ideas in his Meta-Annotations post and even wrote a possible implementation.

It would be convenient to have plain Java SE support to annotation inheritance, abstraction and encapsulation. It’s a major shift in the way annotations are handled by all the technologies out there. This only makes sense if everyone starts supporting this kind of behaviour.

One might ask, do we really need this feature? Let’s try to weigh in some pros and cons:

Pros

Simplified Code.
Annotation Reuse.
Annotation configuration not directly tied to the code. Values could be overridden.

Cons

Another layer of abstraction.
Freedom to create custom annotations could obfuscate the real behaviour.
Possible to run into some kind of multiple inheritance pitfall.

Conclusion

It’s unlikely that this Meta-Annotations are going to be available in Java SE for the foreseeable future. In Java 9, most of the focus is in Jigsaw. We don’t have much information yet on Java 10, other than Value Types and Generic Specialization. In fact, all these annotation concerns are not really a problem for plain Java SE.

The amount of annotations present in our source files is becoming a problem regarding readability and maintainability. This is especially true for Java EE and other similar technologies. Think about HTML and CSS. If you’re developing an HTML page and you only need a couple of CSS styles, you usually inline them in the elements or include them directly in the page. If you start to have too many styles, then you proceed to extract them into an external CSS file and just apply the style. I do believe that something must be done either by adopting Meta-Annotations or something else. Do you have any other ideas? Please share them!

Resources

Don’t forget to check David Blevins post about Meta-Annotations. He explains this much better than me, including the technical details.

Also a presentation JavaOne EJB with Meta Annotations discussing these ideas by David Blevins.

And what better than to listen David Blevins himself? Java EE 7, Infinite Extensibility meets Infinite Reuse.