A lesson in book judging

“Never judge a book by its cover.” – Mill on the Floss, George Eliot, 1860.

Many decades ago, when I was a young teen, a gentle knocking came on our front door and my mother got up to answer it. I peeked around the dining room door to see who it was. A dishevelled old man graced our doorstep, hair like that of a scarecrow, shoes with string for laces, an old jacket around his body and a cloth sack over his shoulder. He asked in almost a whisper “could I have a drop of water?

Several beggars roamed the housing estates in those days and you’d have at least one come knocking in any given week. The smell of alcohol would often announce them before they’d even reached the open front gate. If spotted early they would be shooed away at the window, or given a sharp “not today” before they’d even opened their mouth.

But not this man. I had seen him before and remembered that my mother would speak with him. This time she invited him into the hallway while she went to the kitchen. He saw me at the dining room door and smiled. I was transfixed, and returned a hesitant “hi“.

My mother soon returned with the glass of water and something in a paper bag that she handed to him. A sandwich? Then she pressed something into his palm, which I believe was some money. He took one small sip of the water, said “thank you, that hits the spot“, handed back the glass, saluted his forehead with two nicotine-stained fingers and bade his farewell as my mother closed the door.

I turned to my mother, “he lied about needing the water.

Yes,” she said, “he didn’t need the water, but he needed his dignity. He’s a good man who has fallen on hard times.”

Knowing that other such callers would be rapidly despatched, I asked her “how do you know he’s a good man?” I expected a “just because” kind of answer, but instead she opened the front door and told me to watch. The man was now two or three doors away, standing back from another front door and obviously being told there was nothing for him. He saluted softly and walked back to the front gate, and despite the rejection he closed it gently behind him.

That’s how I know,” she said.

Hello World with JAX-RS on Tomcat 10, Jersey and JDK 17

Sometimes the best place to start a new coding experiment is with “Hello World” (HW), the absolute minimum, often borrowed from someone else who has kindly published it for the benefit of others. The fewer assumptions made by the HW project the more useful it will be to the community. Some HW demos tend to make a lot of assumptions (IDE, build mechanism, dependency management etc.) and this presents problems for those who don’t match.

I recently created a starting point for a JAX-RS API running on Tomcat 10.1 and JDK 17, and when I looked around the Web I noticed that similar demos tended to assume Eclipse, or Maven, or Gradle and so on in the development environment. Here I am going to show how to create a working JAX-RS API endpoint without the aid of an IDE or build mechanism. This is as bare-bones as I can get.

But there are some assumptions.

First up, I’m using the latest (as of 2023) Tomcat 10.1 as a container. As Java-based servers go, this is fairly basic and if you can get something running in Tomcat then you can pretty much get it to work in any container. I’m running this on a recent JDK 17 (this one has long-term support). Obviously I’m using OpenJDK rather than some proprietary/expensive version.

Second, as Tomcat doesn’t come with any JEE goodies, I’m using Jersey to provide the implementation for Jakarta REST (JAX-RS).


I will leave out the exact version numbers of the libraries as you can find the latest versions online quite easy. For example, in my collection of Jars I have jakarta.ws.rs.-api-3.1.0.jar, which is the current latest. You can find these jars in many places, such as on Maven. Wherever you get them, verify their integrity via the usual checksums.

Jakarta libraries

These are mainly the APIs, not the implementation.

  • jakarta.ws.rs-api
  • jakarta.inject-api
  • jakarta.validation-api
  • jakarta.xml.bind-api
  • jakarta.servlet.jsp.jstl-api
  • jakarta.servlet.jsp.jstl (Apache implementation of taglibs)

Jersey libraries

These come from the Glassfish/Eclipse Jersey reference implementation.

  • jersey-common
  • jersey-server
  • jersey-client
  • jersey-container-servlet-core
  • jersey-container-servlet
  • jersey-guava (repackaged Google-authored utility classes)
  • jersey-media-jaxb
  • jersey-hk2 (JSR 330, dependency injection)
  • hk2-api
  • hk2-locator
  • hk2-utils

REST Application

The Web Application will be deployed to http(s)://<domain>/demo and the Jakarta REST application within it will be at sub-path /rest, and it will have one endpoint called “hello” that will accept a GET request containing a ‘key’ parameter, and respond with plain text: “Hello key“. Thus the URL http(s)://<domain>/demo/rest/hello?key=World will return the text: “Hello World”.

Instead of a web.xml, I am using annotation to declare the class that identifies the REST app, and I am not overriding the base class methods so that the default behaviour will be to search the rest of the app classes for annotation indicating endpoint handlers. This is com/example/DemoApp.java:

package com.example;
import jakarta.ws.rs.ApplicationPath;
import jakarta.ws.rs.core.Application;
@ApplicationPath("rest") public class DemoApp extends Application { }

This is the implementation of the “hello” endpoint, com/example/Hello.java:

package com.example;
import jakarta.ws.rs.*; // Consumes,DefaultValue,GET,Path,Produces,QueryParam;
import jakarta.ws.rs.core.MediaType;
import jakarta.ws.rs.core.Response;
@Path("hello") public class Hello {
  public final Response hw(@DefaultValue("key?") @QueryParam("key") String key){
    Response.ResponseBuilder rb =
        .ok("Hello "+key,MediaType.TEXT_PLAIN)
    return rb.build();

Finally, to define the path of the deployment (“/demo”), I need a context.xml file containing this one line:

<?xml version="1.0" encoding="UTF-8"?><Context path="/demo"/>

To compile the source files at the command line producing the necessary class files, use this:

javac -d CLS --sourcepath SRC --cp JARS:TCLIBS --source 17 --target 17 SRC/com/example/*.java


  1. javac is the Java 17 compiler executable, and may be the absolute filename if necessary
  2. CLS is the (existing) directory where the class files will be placed
  3. SRC is the path of the directory containing the com/example/*.java source files
  4. JARS is a list of the 17 Jar files listed above
  5. TCLIBS is the list of the Jar files in Tomcat’s lib directory
  6. Lists are colon:separated in Unix, and semicolon;separated in Windows, so adjust if necessary

The command will produce two files: CLS/com/example/DemoApp.class and CLS/com/example/Hello.class

That’s it. There is no more code. No more configuration files. The above files need to be packaged into a WAR file (a Zip file with a .war extension) containing the following:

  • META-INF/context.xml
  • WEB-INF/
    • classes/
      • com/
        • example/
          • DemoApp.class
          • Hello.class
    • lib/
      • The 17 Jar files go in here

Deploy the demo.war file by copying it to Tomcat’s webapps directory and start Tomcat. Test by using a browser to exercise the following path on your server: /demo/rest/hello?key=World


There are many tools that you can use to make the above process easier. For example, to make the WAR file you can open a command line at a directory containing the pre-filled META-INF and WEB-INF directories, and issue the following command:

jar -cf demo.war

The JDK’s jar tool merely zips the contents recursively to create the WAR file.

You can collect the names of the Jar files into files and use the “@” feature of javac in order to reference these lists instead of having all the files individually in the command line. You can use a build tool to track these dependency lists, or use a similar feature within your preferred IDE. Using tools makes this process much easier than the manual steps I outlined above, though I hope you can now see what is happening under the hood regardless of your chosen tools.

One hundred and fifty to one

I’m in the rather unfortunate position of having to migrate a client’s accumulated data from an old version of MySQL to the latest, and from old hardware to new, in a hosted environment where my only access is via a SQL client, with minimal down time.

Let’s ignore stored procedures, views, triggers etc. Those can be reproduced easy enough. It’s the tables containing millions of rows that will eat up the most time[1]. If you do a cursory check of the manuals and guides you’ll find that the normal approach is to use mysqldump to create a (massive, compressed) SQL file filled with table creation and row insertion operations. You then pump that dump file through a MySQL connection to rebuild the entire database in another server.

To kick off the exercise, I used mysqldump to generate the compressed dump file. It took about as much time as it took me to make and drink a cup of coffee. Checking the head of the resulting dump I could see the usual preamble, a few embedded comments intended to be used by MySQL as hints to adjust some settings during processing, a table CREATE and then line-after-line of INSERT statements. It was near the end of my working day, and I had fired up a test box to receive the dumped data, so I kicked off a restore via a single DB connection and locked my PC for the evening.

It was still running when I logged in the following morning. In fact, it took until lunchtime to complete.

If this were the actual production migration, we would be in serious trouble. Timings like this would likely see the client opt for a migration during anticipated non-usage, such as New Year’s Eve! Not only would that put off the process for several months, it would ruin what should be a day off. This migration has to get down to just a few hours so that it can happen in the very early hours of the morning, in the next few weeks, when an outage of up to 4 hours could be accommodated.

There were several factors taken into consideration to determined the ultimate solution:

  • There would be no other users connected to the old and new database servers during migration.
  • The target server’s hardware can handle up to four concurrent threads of execution without breaking a sweat.
  • Table indices are not needed during table restoration. (They can be added later.)
  • There are several big tables, some with only a few columns, some with many.

Here is what I did:

  • The dumped SQL is piped through a custom process that splits the dump into four.
  • Each split replicates the header and footer from the original dump, but only includes a subset of the tables.
  • The subsets are non-overlapping and chosen so that the total number of rows are spread evenly across the splits. (There’s some adjustment here to account for the fact that some table inserts are more intensive than others because of the width of the tables and other factors. It took a bit of benchmarking to work out the optimum.)
  • The table creation is adapted to defer indexing and other changes appropriate for the migration[2].
  • I’m using the “extended insert” option of mysqldump so that each INSERT statement contains data for multiple rows. These optimized inserts can be up to 67 million characters (max_allowed_packet) for my target server, in theory[3].
  • The four partial dumps are restored in parallel to the target server. Were it not for the numerous restrictions, some of this could be achieved via MySQL shell’s parallel import. As that’s not available, the bespoke approach is needed.

The result:

The restore took just a few minutes. Yes, minutes. In fact, the performance improvement was 150:1

It looks like when the migration happens for real, I might just have time for two coffees :)

[1] Actually, no, we also have to consider the complete audit of the existing application software to ensure that all the old SQL usage was compatible with the new v8, and setting up a replica environment to run the entire suite of unit tests with the new database, while watching for slow queries and other gotchas. That took two weeks, but as a separate activity that has no impact on the expected migration down time.

[2] Interestingly, MySQL 8 has deprecated utf8 in favour of utf8mb3 and will automatically set this charset during table creation when you have specified utf8. So, while you would need to adjust the CREATE statements to change PKs and indices, you can leave the charsets alone.

[3] Using LOAD DATA would be even faster, were it not for the fact that I can’t access the underlying file system of the target server.

Rug pulling

This involves AWS EC2 AMI deployment/setup automation, and if that makes you shudder then look away now.

Last week I was completing some automation that takes a blank EC2 through a carefully scripted sequence of steps to produce a production-ready platform for a specific live service. It’s not Chef, or Puppet or any of a number of config/build automation solutions. It’s just a simple shell script that incrementally adds functionality either to enhance its own configuration/build abilities and/or support the target setup. It’s close enough to the OS to support the granularity of control that I need, while being abstract enough to be reasonably compact. The current script is just shy of a thousand lines.

This script starts with the “reasonable” assumption that it has the minimal functionality provided by the default OS (Amazon Machine Image with Amazon Linux 2, AKA AL2), does a quick “yum update“, mounts drives, defines swap space, adds a repository of very useful tools via “amazon-linux-extras install epel” and continues to add more tools, libraries, directories and much more, through multiple OS reboots where necessary until eventually I have a working system.

Setting up the initial EC2 for testing can be automated so that there are real cloud instances to use during development of this scripted process. However, I have found it far more flexible (and efficient in time and money) to deploy an AL2 instance on a local VirtualBox during this time. This is something that Amazon intentionally supports. The free OS images are available to download and I have my own script that will create new Virtual Machine instances from these images in a few seconds, ready to test-run my platform installation script.

Last week I had reached the point where the entire automation was reliable for all the use cases that were required. Now the testing needed to move from VirtualBox instances to EC2 instances. To do my first scripted installation I needed a fresh EC2 and I decided to manually create one using the AWS console. It only takes a minute to get an instance set up.

This is the point where the rug was pulled from under me.

Having clicked the “Launch Instance” button from the EC2 part of the AWS console, I was presented with the Application and OS Images options, and I expected that the AWS Amazon Linux 2 would be the initial (default) selection. Instead, I was presented with this new default option:

Amazon Linux 2023 AMI
ami-09dd5f12915cfb387 (64-bit (x86), uefi-preferred) / ami-0de2a2552e7fe4b4d (64-bit (Arm), uefi)
Virtualization: hvm   ENA enabled: true   Root device type: ebs


Amazon Linux 2 is now the second option on the list of Amazon Linux variations. While I had been busy creating an installation for AL2 (which I had also tested on RedHat-like environments such as CentOS and Rocky) and using an Amazon-supplied VirtualBox image, they had been busy launching a new version of the OS, Amazon Linux 2023, together with a schedule for the next few years offering a new version every 2 years.

There are a number of differences that I must address:

  • SELinux is now enabled by default. I’m OK with that as I try to make use of whatever security features are present, but as it was not used by default in AL2 it is not obvious if the custom installation will trip up the AL2023 security. I will have to monitor the SELinux logs. (Thankfully it defaults to permissive.) Still, this is wading into unexplored territory.
  • Amazon Linux is no longer compatible with any particular release of Fedora. This could further limit my deployment/development options as I like to work with multiple distros to limit lock-in.
  • The package manager changed from yum to dnf, or if you want to be pedantic: from the modified version of the original Yellowdog UPdater (YUP) known as Yellowdog Updater, Modified (YUM) to the Dandified Yellowdog Updater Modified (DNF). Give me a break! But seriously, while there is a strong family resemblance, replacing yum with dnf is not all plain sailing.
  • EPEL is gone! The Fedora Extra Packages for Enterprise Linux won’t work on AL2023 because of a pile of compatibility issues. AL2 was much the same as CentOS 7, and therefore most of the EPEL packages were compatible, but not so for AL2023. This is an extra headache because a lot of my scripted installs would make use of EPEL.

Dealing with yum/dnf should be easy enough, with the help of a bit of abstraction. Especially if I want to keep some backwards compatibility with the AL2 installations. The loss of EPEL could be a bigger headache.

In the meantime, I have one even bigger (hopefully short-term) headache: despite the written promises, Amazon has not (yet) released an on-prem deployable image for VirtualBox or equivalent. People have been complaining for weeks about this. Until I can get an image to use locally I have no choice but to experiment and develop within the cloud itself.

As for my AL2 deployment that was ready to go, that’s now on hold because of the demotion of AL2. I will be reading the AL202X Release Notes intensively to see what other changes and potential pitfalls I have to deal with before I restart the scripted deployment activity. Nothing like a bit of light reading on a sunny evening…

CRC32C, JDK 17 and Netbeans

Even though the source level of /some/source/path is set to: 17, java.util.zip.CRC32C cannot be found on the system module path:

This one took me hours.

A while ago I migrated a major project to JDK17 and made many adjustments to ensure further development on it could be done in the latest Netbeans 17. As NB17 is bleeding edge, I regularly check the IDE log to make sure it’s not having a hard time. Unfortunately, doing almost any editing of the project caused warnings such as the message at the top of this post to appear. Hundreds of them. Flooding the log.

This didn’t cause any problems with my ANT build on Jenkins, but I noticed it was making Netbeans pause from time to time, and it worried me that this might also affect dev-time diagnostics and so on. I needed to figure out what was causing the logged message, and if I could stop it.

An online search, unfortunately, showed that this problem is just as perplexing to others, and despite some interesting theories nobody could offer an explanation, solution or work-around.

As my instance of Netbeans 17 is compiled from source by me, I had the source at hand to investigate. Tracking down the message to JavacParser.java was easy, but it revealed (over a thousand lines into the file) that the warning is a consequence of the parser deciding that the source should be downgraded to 1.8.

No, can’t be having that!

The next few hours led me down a rabbit hole of configuration nonsense. I was adamant that any fix I was going to determine would not require me rewriting Netbeans, as then either I’d have to apply a patch every time I updated/recompiled the source, or I would have to submit the change to the core Netbeans project. As the latter would mean I’d have to consider possible consequences way outside of my limited use cases, this was not an option.

One thing that puzzled me was the fact that the message ended with “the system module path:” but was followed by a blank. No path was actually identified. Looking at the JavacParser source I could see that the moduleBoot variable was empty. This then led me to more time wasted trying to find ways to set that variable via external configuration, with the hope that if I could do that then I could point it at the JDK 17 modules (specifically the jmods/java.base.jmod file where the CRC32C.class file is located). I did not succeed, so I started climbing out of the rabbit hole in the hope that there might be another approach.

Indeed there was another approach. The key test determining the downgrade to 1.8 accompanied by the warning message was of the form:

!hasResource("java/util/zip/CRC32C", new ClassPath[] {moduleBoot}, new ClassPath[] {moduleCompile, moduleAllUnnamed}, new ClassPath[] {srcClassPath})

I had been concentrating on the new ClassPath[] {moduleBoot} part, mainly because this is what was specifically mentioned in the warning message. However, the logic of the hasResource() method revealed that it was searching for CRC32C.class within the module path or the compile/unnamed paths, but also looking for CRC32C.java within the source class path (srcClassPath). Just to be clear, the CRC32C class is available in the JDK17 modules, and Netbeans should be able to determine this and therefore decide that the project is being developed within and for a JDK17 environment. The test, in fact, looks for CRC32C in order to decide if the source level is at least 9. If that passes, it then goes on to look for java.lang.Record to decide if the source level is at least 15.

So, if it could find the source file (.java) instead of the class file (.class) then the test would pass. Fortunately the source path involved refers to the root(s) of where the project sources are located. So if I were to create a path java/util/zip and place CRC32C.java in there, the test would succeed. But wouldn’t having a copy of CRC32C.java in the project create other problems? It would, if the file actually had anything in it. The test is only looking for the existence of the file. It doesn’t actually have to have a definition of the real class. So I simply added a java/util/zip/CRC32C.java and (for good measure) a java/lang/Record.java to my project, with both files containing this one-line comment:

/* Here to keep Netbeans happy */

I also updated my .gitignore to ensure this hack didn’t get pushed up to the repository.

Did it work? Yes, it worked.

In summary: Netbeans is looking for CRC32C in certain paths to confirm that the source level is at least Java 9, so to ensure that it passes this test I created a dummy (i.e. empty) java.util.net.CRC32C source file in the project, and a similar dummy java.lang.Record source file to ensure it also passes the Java 15 test.