Long Garbage Collection time on processing.. alert.

Recently our J2EE application converted to Springboot architecture. Although I 've mentioned here as Springboot architecture, it is not a complete Springboot powered application. Our front end is not MVC or a Rest service, it is JSF 2.X. We are not using embedded Tomcat, instead standalone Tomcat. 

Having all these in this application,  we ran the show around 2 weeks no issues, a good migration. Great work !!

But one fine day, Boom!. We received Long Garbage Collection alert. Tomcat restarted, everything looks ok, back to work. Nothing specifically found in the application logs. So we thought it is just one isolated case.

After two weeks, alert received again. No dead locks in thread dump. Wasn't able to get the memory dump since it takes at least 30 min to create the dump, where the system is  not available for the customers while obtaining heap dump. Issue was now happening almost daily. Day and midnight where there were no significant load in the system. We knew that this was not related to the load, yet something else triggering. 

Above is the pattern of garbage collection in Edan space (Young Gen). You can see, when the issue triggered, it was continuously calling for long time.

Above is old gen memory. You can see that it was growing slowly, and have started calling garbage collection later. To respect to the data, I had to mask some of the details here. Most important thing is the graph pattern.

Meanwhile, in the tomcat console, we found there were long lasting http sessions in the server. Those were few days old, yet still there in the memory. This was a requirement in the application, but it has this negative effect of long lasting sessions. Fixed this issue, and now no more long lasting sessions. Further, we closed all the Streams, IO/http connections after used, and optimised some of the memory usage in the application, but still issue was occurring.

One day midnight we were able to get the heap dump. The heap dump analysis showed us that there are lot of String objects there in the memory. (No choice, strings are highly used in the application for various logics, for report and email generation, etc.) We used Eclipse MAT and VisualVM to analyse the heap dump. Eclipse MAT showed that there is a memory leakage in the class loader in their leak detection analysis.

Above are from Eclipse MAT.

But interestingly, VisualVM gave us a hint that large objects were held by Jasper report classes.  However, we didn't have an idea how this happened and how to simulate the issue in our DEV or UAT environment as the issue was not occurring always. We further analysed the user behaviour when the issue occurred and found a way to simulate the issue.

Marvellous !!!  The culprit was Jasper reports. When the pdf file does not have enough space to add the string generated in runtime, Jasper report library getting stuck and keep accumulating the memory internally. Allocated more space in the pdf file to include the string, and the issue was fixed. 😊😊😊

(GC Algorithm is G1 GC.)

Few hints I can give,

• Check the sessions in Tomcat console - this will give you long lasting sessions with the timing.
• Check the threads in Tomcat console - this will show you whether any thread is stuck in Serving state. Which could be a problematic thread.
• Check the large objects in VisualVM.
• If you are a customer facing application, then use GA Analytics to see user flow.

Some readings can be found here. 

https://dzone.com/articles/troubleshoot-sudden-cpu-spikes

Ajeewa Attamaka Sil ( Ajeewa Ashtamaka Seelaya )

This post is related to the Buddhism. Every time when I need to observe the Ajeewa Ashtamaka Sil on my off day, I had to go to YouTube and search for it. It is very hard to find these 8 precepts in internet in one place. Moreover, there is no much articles available about Ajeewa Attamaka Sil to learn. Therefore, for the benefit of others, I thought to mention Ajeewa Attamaka Sil precepts in my blog.

For further reading about the meaning of each of these, you have to explore more on the internet. I'm not a right person to explain further meaning of each.

1. Panathipatha veramani sikkhapadan samadiyami. (I undertake the training rule to abstain from taking life. ) 
2. Adinnadana veramaṇi sikkhapadaṃ samadiyami. ( I undertake the training rule to abstain from stealing. )
3. Kamesu micchacara veramaṇi sikkhapadaṃ samadiyami. ( I undertake the training rule to abstain from sexual misconduct. )
4. Musavada veramaṇi sikkhapadaṃ samadiyami. ( I undertake the training rule to abstain from false speech. )
5. Pisunawacha veramani sikkhapadaṃ samadiyami. ( I undertake the training rule to abstain from slandering speech. ) 
6. Parusawacha veramani sikkhapadaṃ samadiyami. (I undertake the training rule to abstain from harsh speech. )  
7. Sampappalapa vermani sikkhapadaṃ samadiyami.  (I undertake the training rule to abstain from frivolous speech. ) 
8. Michcha ajeewa vermani sikkhapadaṃ samadiyami. ( I undertake the training rule to abstain from engaging in wrong livelihood. ) 

Below are the Sinhala meaning of last 4 precept mentioned above, respectively.

• Kelam keemen welakemi
• Parushawachana keemen welakemi 
• Nissara wachana katha kireemen welakemi
• Weradi divi pewethin wen wemi.

May Triple Gem Bless You !!

How to generate alert mail to find Unchecked Exception in your JSF application (ExceptionHandlingActionListener)

Past few days I had a dilemma of how to generate email alert if my JSF application has any RuntimeException. Finally I found a solution.

ActionListenerImpl class   help me for that.

No need to talk, let's start the party.

First step:
We need to create ExceptionHandlingActionListener  class that implements  ActionListenerImpl.
in the ExceptionHandlingActionListener class, we should override the supper class processAction(ActionEvent event) method. And write our email alert code inside the catch block.

So in my sysout " Typing email " section, we can call to a email generation method and pass the StackTrace to email body. Email formatting is not in my radar.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

package com.imesh.test; import javax.faces.event.ActionEvent; import javax.faces.event.ActionListener; import com.sun.faces.application.ActionListenerImpl; public class ExceptionHandlingActionListener extends ActionListenerImpl implements ActionListener { @Override public void processAction (ActionEvent event) { try { super.processAction (event); } catch (Throwable e) { e.printStackTrace(); try { System.out.println("----------------------- SYSTEM EXCEPTION EMAIL SEND BEGIN. -----------------------"); System.out.println("Typing email"); System.out.println("----------------------- SYSTEM EXCEPTION EMAIL SEND END. -----------------------"); } catch (Exception f) { System.out.println("----------------------- SYSTEM EXCEPTION EMAIL NOT ABLE TO SENT. -----------------------"); f.printStackTrace(); } } } }

Second step : 
Add this listener to the faces-config.xml

1 2 3 4

<application> <action-listener> com.imesh.test.ExceptionHandlingActionListener </action-listener> </application>

Awesome!!!,  We are done. Try create any RuntimeException in your application. Magically it will handle by our ExceptionHandlingActionListener  and generate the email for you.

Thank you, Comments are welcome !!.

My experience with Qistina Express bus to Kuala Lumpur

Qistina Bus Review

I have seen many places that Qistina has bad review on certain things. I traveled to Kuala Lumpur from Singapore on 6th Feb 2016.

My depart time was 8.30 am at Little India Arcade.
Since I read all the review previous day, I just did another search to see whether the bus has got any seats available, thinking that if it's full then no need to wait for more pax. The bus was not appear in the search results. So I thought the bus is full and can go without waiting it to get filled.

As most of the travelers mentioned, they were late. But the office was opened. I saw some ppl said that office was not opened till 10.30 am. But by the time I reach 8.30, the counter was already opened.

Bus came around 9 am luckily. But it was waiting there till 9.30 for more passengers. There is no seat preference as we book. You can sit freely anywhere you need. So they add ppl who ever wants to go by this bus. Bus is full now.

Then the journey started it's a big traffic due to chinese new year. After JB check point, the bus stopped in a petrol shed. Not to fill the petrol. I don't know where the hell driver went but the bus was stopped there around 40 mins for nothing. We did not move to another bus but we had to waste the time for nothing.

Bus is not smelly, Driver didn't smoke during travel, Seats are average. A/C also OK. Driver was waiting for all passengers. He is fast but safe drive.

So this is my experience the Journey to KL.

Sri Maju Group reivew

On the way back to Singapore, I used Sri Maju travels.  I had perfect journey. The bus reached on time to the TBS point and depart within 5 mins. No stops no wait. The driver stopped after 1,5 hrs for a small break in a petrol shed. again started and stopped in the usual stop for a tea break.

So we started 1.30 from TBS we reach JB 6 pm. Amazing. The drive was very nice, he saw there is a block in the one enterence to the JB check point then he turn the bus in to another way. No traffic we went directly to the check point. And he wait until everyone come on each checkpoint and each time he count the pax.

It's a Scania bus. The journey was not tired at all. Price was 18 SGD (not bad at all huh?)
We reach around 7.20 to the Golden Mile Complex.

How ever I will not use the Qistina Express again but for sure Sri Maju. Thumbs up  for Sri Maju !!! 

Damro Innovex Electronic goods and after service.

I would like to write the review of the Damro Innovex  Pedestal Fans.

Sometimes ago I bought two fans (one after another) from Damro which is Innovex brand. After one and half year my first fan suddenly stopped working. So I gave to the Damro Dalugama showroom to check. I am out of warranty. They took almost 2 weeks and came back with  Rs 900.00 service fee. When I asked them what did they do to fix, the showroom staff said they have no idea what the service center did.
So I had my fan back. But after a one week again the fan is stopping intermittently. Then again it worked for few weeks with a bad sound and stopped working again. I went back to Damro, told them that few weeks ago I got repaired this but doesn't work again. They said they have service warranty for 6 months, so let's send this back to service center.

After 2 weeks, they phoned me and said the fan is fixed but I was charged Rs 1200.00 for the fix. I said what the hell you said it has 6 months warranty but now you charge me again Rs 1200.  Then the staff said they will check with service center again but still didn't comeback.

My loyalty for Damro is fading now. I had few electrical items from them thinking that I am supporting to Sri Lankan owned business.

I just want to tell that based on these info decide whether you still go with Damro Innovex product and service or pay bit more higher price and purchase a long lasting branded product.

I checked around and found that KDK is quite long lasting. Although it's double the price as Innovex fans, it could be a life time asset for you.

PC Dreams Laptop Hospital - Singapore

Few weeks ago suddenly my laptop screen became white. So I checked in web and found a place call PC Dreams in Funan mall.
http://forums.hardwarezone.com.sg/notebook-clinic-77/pc-dreams-best-laptop-repair-shop-2257096-167.html

They hold the laptop by charging 64$ for inspection.  And after three days phoned me and said it would cost 240$ to replce thr screen. The laptop it self is only cost me 500$. I checked with Acer service center.  They said the screen cost around 80$ only.

So I was misleeding with hardware zone forum and paid 64$ for nothing. Also they have broken the screem arm which I saw only I came home.

Resource in Maven POM

Recently I was engaged to a project which convert a legacy ant build application into Maven build. My main application project has several sub projects. One child project fully contained sql scripts and cron job scripts. So I had to find where to include these sql and .sh files in the maven structure. I google and went through few articles to find out the correct place.

There is no clear cut solution I found. Some people suggest to put it into resources folder and some suggest to use separate folder. So, I thought to go ahead with a separate folder because the content of resource folder will attach to ear once I build it ( but still we can exclude them. Yet I thought it could be messy).


Resources

While surfing the web, I found more about Resources. We can include our separate folder into jar file. In my case, I do not want sql and .sh files to available in final jar. I just want to keep those in the same project. Let's see where to include those files and how to bundle our own folder into a jar.

Below is the apache introduction about Resources..
" Resources is a nice feature of build elements is specifying where resources exist within the project. Resources are not (usually) code. They are not compiled, but are items meant to be bundled within your project or used for various other reasons, such as code generation"

So I included all my sql and .sh files in its own directory instead put them in src/main/resource/scripts/sql, Now my sql and .sh files are located in src/main/scripts/sql and src/main/scripts/sh folders respectively. When I build the project, scripts folder is not included to jar.

Say, now I need to build a jar including the sql and sh files.

For this I used JAR plugin to bundle the resource correctly, and I would specify resources similar to the following:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <build>
  ...
  <resources>
   <resource>
    <targetPath>scripts/sql</targetPath>
    <filtering>false</filtering>
    <directory>${basedir}/src/main/scripts</directory>
    <includes>
     <include>synctableA.sql</include>
    </includes>
    <excludes>
     <exclude>**/*.properties</exclude>
    </excludes>
   </resource>
  </resources>
  <testResources>
   ...
  </testResources>
  ...
 </build>
</project>

See below detail about each tag.

• resources: is a list of resource elements that each describe what and where to include files associated with this project.
• targetPath: Specifies the directory structure to place the set of resources from a build. Target path defaults to the base directory. A commonly specified target path for resources that will be packaged in a JAR is META-INF.
• filtering: is true or false, denoting if filtering is to be enabled for this resource. Note, that filter *.properties files do not have to be defined for filtering to occur - resources can also use properties that are by default defined in the POM (such as ${project.version}), passed into the command line using the "-D" flag (for example, "-Dname=value") or are explicitly defined by the properties element. Filter files were covered above.
• directory: This element's value defines where the resources are to be found. The default directory for a build is ${basedir}/src/main/resources.
• includes: A set of files patterns which specify the files to include as resources under that specified directory, using * as a wildcard.
• excludes: The same structure as includes, but specifies which files to ignore. In conflicts between include and exclude, exclude wins.
• testResources: The testResources element block contains testResource elements. Their definitions are similar to resource elements, but are naturally used during test phases. The one difference is that the default (Super POM defined) test resource directory for a project is ${basedir}/src/test/resources. Test resources are not deployed.

Hope this would help to increase your knowledge about maven resources.