Everyone has good stories about releases that went wrong, right? I’m no exception and I have a few good ones under my development career. These are usually very stressful at the time, but now me and my teammates can’t talk about these stories without laughing.
History
I think this happened around 2009. Me and my team had to maintain a medium to large legacy web application with around 500 k lines of code. This application was developed by another company, so we didn’t have the code. Since we were in charge now and needed the code to maintain it, they handed us the code in a zip file (first pointer that something was wrong)!
Their release process was peculiar to say the least. I’m pretty sure there are worst release procedures out there. This one consisted in copying the changed files (*.class, *.jsp, *.html, etc) to an exploded war folder on a Tomcat server. We also had three environments (QA, PRE, PROD) with different application versions and no idea which files were deployed on each. They also had a ticket management application with attached compiled files, ready to be deployed and no idea of the original sources. What could possibly go wrong here?
The Problem
Our team was able to make changes required by the customer and push them to PROD servers. We have done it a few times successfully, even with all the handicaps. Everything was looking good until we got another request for additional changes. These changes were only a few improvements in the log messages of a batch process. The batch purpose was to copy files sent to the application with financial data input to insert into a database. I guess that I don’t have to state the obvious: this data was critical to calculate financial movements with direct impact on the amounts paid by the application users.
After our team made the changes and perform the release, all hell went loose. Files were not being copied to the correct locations. Several data duplicated in the database and the file system. Financial transactions with incorrect amounts. You name it. A complete nightmare. But why? The only change was a few improvements in the log messages.
The Cause
The problem was not exactly related with the changed code. Look at the following files:
BatchConfiguration
Java
1
2
3
publicclassBatchConfiguration{
publicstaticfinalStringOS="Windows";
}
And:
BatchProcess
Java
1
2
3
4
5
6
7
8
9
10
11
12
13
publicclassBatchProcess{
publicvoidcopyFile(){
if(BatchConfiguration.OS.equals("Windows")){
System.out.println("Windows");
}elseif(BatchConfiguration.OS.equals("Unix")){
System.out.println("Unix");
}
}
publicstaticvoidmain(String[]args){
newBatchProcess().copyFile();
}
}
This is not the real code, but for the problem purposes it was laid out like this. Don’t ask me about the why it was like this. We got it in the zip file, remember?
So we have here a variable which sets the expected Operating System and then the logic to copy the file is dependant on this. The server was running on a Unix box so the variable value was Unix. Unfortunately, all the developers were working on Windows boxes. I said unfortunately, because if the developer that implemented the changes was using Unix, everything would be fine.
Anyway, the developer changed the variable to Windows so he could proceed with some tests. Everything was fine, so he performs the release. He copied the resulting BatchProcess.class into the server. He didn’t bother about the BatchConfiguration, since the one on the server was configured to Unix right?
Maybe you already spotted the problem. If you haven’t, try the following:
Copy and build the code.
Execute it. Check the output, you should get Windows.
Copy the resulting BatchProcess.class to an empty directory.
Execute this one again. Use command line java BatchProcess
What happened? You got the output Windows, right?. Wait! We didn’t have the BatchConfiguration.class file in the executing directory. How is that possible? Shouldn’t we need this file there? Shouldn’t we get an error?
When you build the code, the java compiler will inline the BatchConfiguration.OS variable. This means that the compiler will replace the variable expression in the if statement with the actual variable value. It’s like having if ("Windows".equals("Windows"))
Try executing javap -c BatchProcess. This will show you a bytecode representation of the class file:
You can confirm that all the variables are replaced with their constant values.
Now, returning to our problem. The .class file that was copied to the PROD servers had the Windows value set in. This messed everything in the execution runtime that handled the input files with the financial data. This was the cause of the problems I’ve described earlier.
Aftermath
Fixing the original problem was easy. Fixing the problems caused by the release was painful. It involved many people, many hours, pizza, loads of SQL queries, shell scripts and so on. Even our CEO came to help us. We called this the mUtils problem, since it was the original java class name with the code.
Yes, we migrated the code to something manageable. It’s now on a VCS with a tag for every release and version.
Are you looking for ways to be more productive? It shouldn’t be a secret that performing actions using the keyboard instead of the mouse will save you time. If you think only about a single action it’s not a big deal. What if you use the same action multiple times a day? If you add up all these actions, they can have a great impact on your productivity.
I’m more or less used to drive most of my actions with keyboard shortcuts. When I was younger, I played semi-professionally Real Time Strategy computer games, including Starcraft and Warcraft III. Starcraft, popularized the term APM (Actions per Minute), which counted the number of actions that a player performed per minute. By using tools, it was possible to record APMs and tell if players were using mouse actions or a keyboard and mouse action combination. Usually, players with a keyboard and mouse combination gameplay had a better chance of winning games than the one that just clicked.
So, what does this have to do with code and IntelliJ? Well, I believe that you can increase your code development productivity by learning and using the keyboard shortcuts to perform the desired actions. You can check the keyboard shortcuts on IntelliJ and you can also check the Productivity Guide which monitors your most used actions. This information is very useful, but it may be a little difficult to change your habits right away. To help you with this, I will describe my most used shortcuts in IntelliJ. You can start familiarize yourself with these and slowly introduce additional shortcuts.
Shortcut
Description
CTRL + W / CMD + W
Syntax Aware Selection
This allows you to select code with context. Awesome when you need to select large blocks or just specific parts of a piece of code. If you have this code:
And place the cursor in the auctionFile and use the shortcut, it will select auctionFile. Press it again and the selection will expand to auctionFile -> createAuctionFile(realm, auctionFile). If you press it again, now the selection will expand to files.getFiles().forEach(auctionFile -> createAuctionFile(realm, auctionFile)). Pressing a final time, you get the full piece of code selected.
If you combine it with SHIFT, you can unselect by context as well.
CTRL + E / CMD + E
Recent Viewed Files
This will show you a popup with all the recent files that you have opened in the IDE. If you start typing, you can filter the files.
CTRL + SHIFT + E / CMD + SHIFT + E
Recent Edited Files
Same as Recent Viewed Files, but only shows you the files that you’ve actually changed.
This will try to complete your current statement. How? By adding curly braces, or semicolon and line change. For instance, if you have the following statement:
System.out.print()
Press the shortcut once to add an ending semi-colon. Press it again to add a new line and to position the cursor aligned with the last line.
Another example: if (condition == true)
Press the shortcut to add opening and closing curly braces, and place the cursor inside the if body with additional indentation.
This one allows you to search by name for a Java file in your project. If you combine it with SHIFT, it searches any file. Adding ALT on top of that it searches for symbols. In the search area, you can use CamelHumps (type the capital letters only of the class name) notation to filter files.
I didn’t mention it before, but I guess you are familiar with auto complete via CTRL + SPACE or CMD + SPACE. If you add a SHIFT you get the smart completion. This means that the IDE will try to match expected types that suit the current context and filter all the other options.
CTRL + ALT + ← / CMD + ALT + ←
Navigate Back
This allows you to navigate back like a browser action. It remembers where your cursor was positioned and navigates back even to other files.
Place the cursor in a element and after pressing the cursor the IDE will highlight all the occurrences of the selected element.
There are many more keyboard shortcuts. Almost every action has an equivalent shortcut. It’s hard to learn them all, it takes time and practice. I still learn new things every week, and if for some reason I don’t code as much for a few days, I forget about the new shortcuts I’ve learned. It’s practice, practice, practice! Try to learn a few and master them instead of trying to do everything in one go. It’s easier!
An IntelliJ plugin exists to tell you with shortcuts you should use if you use the mouse. Its Key Promoter, but unfortunately it seems it’s not maintained anymore. Maybe I can update it for the latests IntelliJ versions. I would also like to see in the Productivity Guide a count of actions performed by shortcuts or mouse. If I find some free time, maybe I can do it too.
The batch purpose is to download the World of Warcraft Auction House’s data, process the auctions and extract metrics. These metrics are going to build a history of the Auctions Items price evolution through time. In Part 1, we already downloaded and inserted the data into a database.
The Application
Process Job
After adding the raw data into the database, we are going to add another step with a Chunk style processing. In the chunk we’re are going to read the aggregated data, and then insert it into another table in the database for easy access. This is done in the process-job.xml:
process-job.xml
XHTML
1
2
3
4
5
6
7
<step id="importStatistics">
<chunk item-count="100">
<reader ref="processedAuctionsReader"/>
<processor ref="processedAuctionsProcessor"/>
<writer ref="processedAuctionsWriter"/>
</chunk>
</step>
A Chunk reads the data one item at a time, and creates chunks that will be written out, within a transaction. One item is read in from an ItemReader, handed to an ItemProcessor, and aggregated. Once the number of items read equals the commit interval, the entire chunk is written out via the ItemWriter, and then the transaction is committed.
ProcessedAuctionsReader
In the reader, we are going to select and aggregate metrics using database functions.
For this example, we get the best performance results by using plain JDBC with a simple scrollable result set. In this way, only one query is executed and results are pulled as needed in readItem. You might want to explore other alternatives.
Plain JPA doesn’t have a scrollable result set in the standards, so you need to paginate the results. This will lead to multiple queries which will slow down the reading. Another option is to use the new Java 8 Streams API to perform the aggregation operations. The operations are quick, but you need to select the entire dataset from the database into the streams. Ultimately, this will kill your performance.
I did try both approaches and got the best results by using the database aggregation capabilities. I’m not saying that this is always the best option, but in this particular case it was the best option.
During the implementation, I’ve also found a bug in Batch. You can check it here. An exception is thrown when setting parameters in the PreparedStatement. The workaround was to inject the parameters directly into the query SQL. Ugly, I know…
ProcessedAuctionsProcessor
In the processor, let’s store all the aggregated values in a holder object to store in the database.
Since the metrics record an exact snapshot of the data in time, the calculation only needs to be done once. That’s why we are saving the aggregated metrics. They are never going to change and we can easily check the history.
If you know that your source data is immutable and you need to perform operations on it, I recommend that you persist the result somewhere. This is going to save you time. Of course, you need to balance if this data is going to be accessed many times in the future. If not, maybe you don’t need to go through the trouble of persisting the data.
ProcessedAuctionsWriter
Finally we just need to write the data down to a database:
If you remember a few details of Part 1 post, World of Warcraft servers are called Realms. These realms can be linked with each other and share the same Auction House. To that end, we also have information on how the realms connect with each other. This is important, because we can search for an Auction Item in all the realms that are connected. The rest of the logic is just simple queries to get the data out.
During development, I’ve also found a bug with Eclipse Link (if you run in Glassfish) and Java 8. Apparently the underlying Collection returned by Eclipse Link has the element count set to 0. This doesn’t work well with Streams if you try to inline the query call plus a Stream operation. The Stream will think that it’s empty and no results are returned. You can read a little more about this here.
Interface
I’ve also developed a small interface using Angular and Google Charts to display the metrics. Have a look:
In here, I’m searching in the Realm named “Aggra (Português)” and the Auction Item id 72092 which corresponds to Ghost Iron Ore. As you can see, we can check the quantity for sale, bid and buyout values and price fluctuation through time. Neat? I may write another post about building the Web Interface in the future.
Resources
You can clone a full working copy from my github repository and deploy it to Wildfly or Glassfish. You can find instructions there to deploy it:
