Wednesday, December 14, 2005

Eliminating session persistence on Tomcat shutdown

Thanks to this post, adding:

<Manager pathname=""/>

to the DefaultContext will disable Tomcat's default behavior of saving the session upon shutdown.

lousy, this didn't work.

Friday, November 11, 2005


So, the truth comes out. Oracle and MySQL appear to not hand back TIMESTAMP as millisecond values. After numerous tests (all Java) in multiple timezones we've decided we're not going insane. The databases appear to not deliver TIMESTAMP as milliseconds, but as a String, always. Even from native console applications. There may be milliseconds that reside underneath, but they represent the String and cannot be retrieved. So, no, the TIMESTAMP going is not *always* normalized to UTC (could only get that to be the case when the db box was UTC and the JVM connecting via JDBC was also UTC). We tried multiple scenarios connecting in different timezones and continually found ourselves asking why the date appeared coming back differently in all places if it was really just stored in milliseconds. Perhaps it is, but it's not being given back that way. And the String is transformed by the JVM into the appropriate milliseconds to represent the toString (on the java.util.Date/java.sql.Timestamp) the JVM thinks it should show in your timezone.

At this point the question becomes, "So, how does this work with internationally replicated/distributed databases servicing multiple international JVMs?" It doesn't work, and after several days of testing, we're feeling confident enough to conclude this: the millis are (sometimes) being adjusted by the JDBC drivers to show you what it thinks you expect--so if you're really expecting UTC, then all the boxes' timezones should be GMT/UTC! That's sort of an issue if you plan on using servers in multiple places throughout the world to handle multiple web applications--which is exactly what we cannot assume.

So, the solution is simple:store the milliseconds as a long. Since Java (and most other sane languages) calculates date based upon the Unix Epoch, it makes sense. Furthermore, there's no guess work as to db timezone, connection timezone, driver millisecond manipulation--it's not a concern anymore.

When does the format of a date being storing in the database really matter? Depends on the need for precision, depends on the need of the application and it also depends on the requirement driving it in the first place. When the requirement dictates international interaction, the first thing that should come to mind is to store the date in a common time: GMT. Other consideration should be given to viewers and manipulators, what do they expect to see the time as? Locally or universal? If the answer for all of these questions point to an international need, the above solution is a simplistic and intuitive approach to resolving the design decision.

-------------begin Hibernate UserType-------------

import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Timestamp;
import java.sql.Types;
import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Comparator;
import java.util.Date;
import java.util.TimeZone;
import net.sf.hibernate.Hibernate;
import net.sf.hibernate.HibernateException;
import net.sf.hibernate.UserType;

* Custom Hibernate type for precise UTC millisecond persistance and retrieval using a Timestamp.
* All values represent the given time from the Unix Epoch, 1 Jan 1970 00:00:00.000
* See for more info.
* Credit: Rob @ "
public class UtcTimestamp extends Timestamp implements UserType {
* Gets the value of SQL_TYPES
* @return the value of SQL_TYPES
public static int[] getSQL_TYPES() {
return UtcTimestamp.SQL_TYPES;

* SQL type.
private static final int[] SQL_TYPES = { Types.BIGINT };

* Creates a new UtcTimestamp instance at this moment.
public UtcTimestamp(){

* With the given milliseconds.
* @param utcMillis a long value
public UtcTimestamp(long utcMillis){

* Make a copy of the Timestamp.
* @see UserType#deepCopy(java.lang.Object)
public Object deepCopy(Object obj) throws HibernateException {
return (obj == null) ? null : new Timestamp(((Timestamp)obj).getTime());

* Compare via {@link Object#equals(java.lang.Object)}.
* @see UserType#equals(java.lang.Object, java.lang.Object)
public boolean equals(Object x, Object y) {
return (x == null) ? (y == null) : x.equals(y);

* Timestamps are mutable.
* @see net.sf.hibernate.UserType#isMutable()
public boolean isMutable() {
return true;

* Return an instance of the Timestamp or null if no value is specified.
* @see net.sf.hibernate.UserType#nullSafeGet(java.sql.ResultSet, java.lang.String[], java.lang.Object)
public Object nullSafeGet(ResultSet rs, String[] columns, Object owner)
throws HibernateException, SQLException {

long value = rs.getLong(columns[0]);
Timestamp timestamp;

if(rs.wasNull()) {
timestamp = null;
} else {
timestamp = new UtcTimestamp(value);
return timestamp;


* Set an instance of the Timestamp into the database field.
* @see net.sf.hibernate.UserType#nullSafeSet(java.sql.PreparedStatement, java.lang.Object, int)
public void nullSafeSet(PreparedStatement statement, Object value, int index)
throws HibernateException, SQLException {

if(value == null) {
statement.setNull(index, Types.BIGINT);
} else {
Timestamp timestamp = (Timestamp)value;
statement.setLong(index, timestamp.getTime());

* Return the {@link Timestamp} class.
* @see net.sf.hibernate.UserType#returnedClass()
public Class returnedClass() {

* Return the supported SQL types.
* @see net.sf.hibernate.UserType#sqlTypes()
public int[] sqlTypes() {
return SQL_TYPES;

* All dates return their value in GMT time.
* @return a String value
public String toString(){
SimpleDateFormat dateFormat = new SimpleDateFormat(DateUtils.DATE_FORMAT);


return dateFormat.format(this);

* Obtain a UtcTimestamp from a Date.
* @param date a java.util.Date value
* @return an UtcTimestamp value
public static UtcTimestamp valueOf(java.util.Date date){
UtcTimestamp utcTimestamp = new UtcTimestamp(date.getTime());

return utcTimestamp;


-------------end Hibernate UserType-------------

External firewire dvd burner

I have a Toshiba SD-R5372 that I wanted to use to backup my system. Didn't take too much effort to get it working last night. I added SCSI CDROM support as a module and also 1394 SBP as a module (the other 1394 options I already had built-in to the kernel).

Emerged the latest k3b, reboot, modprobe sbp2 and modprobe sr_mod.
Plugged in the burner and watched /var/log/messages recognize it.
Added a device in k3b as /dev/sr0
Burned 4G in 20 minutes.

Once the burning was complete I mounted the drive as "mount /dev/sr0 /mnt/dvdrw" and verified the disc has my files!

Wednesday, November 09, 2005

Vonage, how easy

Number transfer took 20 days exactly. Called up AT&T and the phone service was shutoff today. Walked out side, opened up the panel, disconnect my wires (light blue and white blue for me). Walked in, plugged the phone line from the Vonage router into the wall. Picked up the phone plugged into the wall downstairs and have been calling out (and receiving) ever since. Sweet.

Secondly, Melissa has been talking to family in Idaho while I'm downloading two Sun apps @ 245KBps! Wow.

Further confirmation all dates are UTC millis, regardless of db timezone settings

(08:36:39) Russ: have a second?
(08:36:51) jamisbuck: sure
(08:36:57) Russ: do you use Mysql much?
(08:37:02) Russ: or Oracle?
(08:37:05) jamisbuck: mysql
(08:37:07) jamisbuck: I used oracle at byu
(08:37:17) Russ: great; done much with timestamps in mysql?
(08:37:22) jamisbuck: no, I avoid them
(08:37:26) jamisbuck: they don't act very intuitively
(08:37:36) Russ: intuitively...meaning?
(08:38:00) jamisbuck: updating the row automatically updates the timestamp, even if it was not explicitly requested
(08:38:03) jamisbuck: which goes counter to my way of thinking
(08:38:12) Russ: ah, right, only for the first column though ;)
(08:38:19) Russ: so, that aside
(08:38:45) Russ: do you explicitly set the timezone of your Mysql server to something other than the default SYSTEM?
(08:38:51) jamisbuck: no
(08:39:15) jamisbuck: we always store the times as utc, and explicitly transform them ourselves to the customer's time zone on display
(08:39:25) Russ: okay, that's exactly what I wanted to chat with you about
(08:39:30) Russ: the concept of storing them as UTC
(08:39:52) Russ: isn't *any* timestamp (or date/datetime for that matter) stored in UTC as millis from the epoch?
(08:40:26) jamisbuck: I have no idea :( but I'm pretty sure mysql dates/datetimes aren't limited to the epoch
(08:40:44) Russ: correct, they're not, the negative values work retroactively
(08:41:49) Russ: I'm struggling with storing time as UTC. Given a Calendar object in Java for today 9 Nov 00:00:00.000 GMT, I can store that in the database as my timestamp and verify the millis going in are the millis coming out--which is what I would expect if the date is indeed stored in UTC.
(08:42:07) Russ: The kicker is, any time I view that date via mysql console (or otherwise), the String representation is MST
(08:42:15) Russ: that is, 8 Nov 17:00:00:00.000
(08:42:30) Russ: not such a big deal, except that it can be confusing to convince that the date is correctly stored
(08:42:35) Russ: when not evaluating the millis
(08:42:35) jamisbuck: what is the timezone for your system?
(08:42:39) Russ: MST7MDT
(08:42:43) jamisbuck: yah, ours is utc
(08:42:52) Russ: I even changed that
(08:42:53) jamisbuck: I bet mysql is converting the date to the system's zone on output
(08:42:57) Russ: that's what I'm thinking
(08:43:10) Russ: I changed everything I could find, *except* my local timezone, and I get the same result
(08:43:20) Russ: even remote connections, from a different time zone
(08:43:36) Russ: which again points to the mysql server on my box "translating" the appropriate time
(08:43:49) jamisbuck: right
(08:43:50) Russ: so, theoretically, if I change my system timezone to UTC, I'll see the String I expect?
(08:44:01) jamisbuck: let me check something quick
(08:44:18) Russ: thanks for your input on this, I've been about on the verge of insanity of verification of this
(08:45:16) Russ: and the only remaining reason is that my system timezone is involved in the String representation
(08:45:31) jamisbuck: ok, I take it back, our servers have CST as the time zone, and mysql is configured to use SYSTEM as the time zone
(08:45:41) jamisbuck: but when I select a date via the mysql console, it doesn't show me any TZ information
(08:45:46) Russ: correct
(08:45:50) jamisbuck: 2004-04-11 07:07:37
(08:46:06) Russ: which if you get the millis, and view the UTC representation, will be the offset of CST
(08:46:22) jamisbuck: k, well, it has never bitten us in our rails apps
(08:46:29) jamisbuck: the times go in and come out correctly
(08:46:32) Russ: exactly
(08:46:41) Russ: as long as you use the millis, and not the string reported, that should be the case
(08:46:52) jamisbuck: to be honest, I use script/console more than I use mysql
(08:47:14) Russ: don't mean the console mysql executable?
(08:47:34) jamisbuck: I use my rails' app's script/console, istead of the mysql client program
(08:47:37) Russ: gotcha
(08:48:15) Russ: well thanks a bunch for your confirmation--my expectation is that whatever driver is managing the connection and result sets returns the millis any time a date/time/timestamp is retrieved
(08:48:37) Russ: the String representation is the secondary, because of the TZ offset calculations
(08:48:47) jamisbuck: alright, glad I could help
(08:48:48) Russ: which brings me around to, why set the timezone on a database?
(08:49:25) jamisbuck: I seem to remember that you can set some properties on the client connection. Maybe some of those specify how to deal with time zones
(08:49:38) jamisbuck: but it doesn't seem like it would help much if you are dealing with dates in multiple time zones
(08:49:44) Russ: exactly
(08:50:00) Russ: in Java you can muck with the millis going in if you provide the connection timezone
(08:50:18) Russ: and the JDBC driver will essentially "translate" the GMT millis to represent the timezone equivalent
(08:50:20) Russ: if that makes sense
(08:50:24) jamisbuck: yah
(08:50:29) Russ: anyhow, thanks again
(08:50:38) Russ: I wondered how Ruby dealt with it
(08:50:50) Russ: and it you've definitely confirmed my expectation
(08:50:52) Russ: have a good one!
(08:50:57) jamisbuck: you too!

GMT time and my headache

So say you care about a given date, like today, 9 Nov. It means so much to you that you want to store it in the database. Turns out if you're using Java, then what you're really storing is the GMT millis of the date and the database is storing that date in the same format. In other words, regardless of the time zone in which you reside, or where the database resides, 9 Nov GMT is 9 Nov GMT.

Regardless of my posts earlier about storing GMT dates in the database, I'm not so conviced that we *weren't* doing that in the first place. Fact is, using the Calendar in the PreparedStatement really only hoses with the millis being stored and my verification that "the right timestamp" was being stored was purely based on the String representation. In all actuality, if millis going in == millis coming out, then the date that is being stored at the moment indicated by the millis is very much indeed being stored in GTM.

But what about the settings for getting Oracle (alter database time_zone=xyz) or Mysql (mysqld_safe --default-time-zone=xyz)? I dunno what bearing that actually has. What I do know is that regardless of the time zone that Mysql is initiaiized with, the String representation of any date always appears as MST. For example, I have a date representing today (9 Nov 00:00:00.000 GMT). I persist this via JDBC and then immediately request back the value of the timestamp. Both sets of millis are identical. However, the date shown from toString() shows 8 Nov 17:00:00.000 since Java knows that JVM's timezone is MST7MDT. Likewise, if I access that same record and look at the date through the mysql command console, I see the same date!

What does this mean? I can explain the Java representation since it's using the timezone of my box (and Bryan and I tested it when he set his Apple to Tokyo time and we verified millis were correct, but String was appropriate for the Tokyo timezone), but I cannot explain the mysql console. Furthermore, why can't I just see millis? Something is translating those millis into MST7MDT. Still need to test this on Oracle. But, for the time being, I'm quite confused about the notion of storing timestamps in GMT when it appears that the Oracle/Mysql are doing exactly that anyway (as indicated by my tests, which I should post).

Tuesday, November 08, 2005

Oracle 9i GMT dates

Oracle made it too easy, here's the test code producing desired results:

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.Timestamp;
import java.util.Calendar;
import java.util.TimeZone;
import java.sql.ResultSet;

public class TestOracleDate{

public static void main(String[] args){

SQLException exception = null;
Connection con = null;
String connectionString = "jdbc:oracle:thin:@testbox:imstest";
String user = "testuser";
String password = "testpassword";
String database = "testdb";
String uuid="c93d4654-40c2-11da";
String identifier="hoser date test";
String driverClassName="oracle.jdbc.driver.OracleDriver";

}catch(Exception ex){ex.printStackTrace();}
con = DriverManager.getConnection(connectionString,user,password);
String insertString = "INSERT INTO TEST_TABLE (cmn_rswd_identifier,cmn_rswd_statusid,cmn_rswd_mdate,cmn_rswd_cdate,cmn_rswd_uuid) VALUES ('"+identifier+"',1,?,?,'"+uuid+"')";
String deleteString = "DELETE FROM CMN_RESERVED_WORD where cmn_rswd_identifier='"+identifier+"'";
//insert = con.createStatement();
final PreparedStatement insert = con.prepareStatement(insertString);
final PreparedStatement delete = con.prepareStatement(deleteString);
System.out.println("executing: " + deleteString);
System.out.println("executing: " + insertString);
TimeZone GMT = TimeZone.getTimeZone("GMT-00:00");
TimeZone MST = TimeZone.getTimeZone("MST7MDT");
System.out.println("First timestamp param, using :" + GMT.getID());
System.out.println("First timestamp param, using :" + MST.getID());
insert.setTimestamp(1,new Timestamp(System.currentTimeMillis()),Calendar.getInstance(GMT));
insert.setTimestamp(2,new Timestamp(System.currentTimeMillis()),Calendar.getInstance(MST));


String queryString = "SELECT * FROM CMN_RESERVED_WORD where cmn_rswd_uuid='"+uuid+"'";
Statement query = con.createStatement();
ResultSet rs = query.executeQuery(queryString);
System.out.println("cmn_rswd_mdate (positional param 1) " + rs.getTimestamp("cmn_rswd_mdate"));
System.out.println("cmn_rswd_cdate (positional param 2) " + rs.getTimestamp("cmn_rswd_cdate"));

catch(SQLException ex){
catch (Exception e) {
throw new RuntimeException(e.getMessage());
if(con != null) con.close();
}catch(Exception ex){ex.printStackTrace();}


GMT dates for Mysql (MST7MDT) via JDBC, success

Finally able to store MST dates as GMT time in the database. Had to load the timezone tables first, /usr/local/mysql/bin/mysql_tzinfo_to_sql /usr/share/zoneinfo/ | /usr/local/mysql/bin/mysql -u root -p mysql. Here's the setup:

mysql 4.1.3-beta-standard

Connector/J:3.1.11-bin OR 3.2.0-alpha-bin


mysql executable: /usr/local/mysql/bin/mysqld_safe --user=root --default-time-zone=MST7MDT &

Prepared statement (Hibernate)code:

public void nullSafeSet(PreparedStatement st, Object value, int index) throws HibernateException, SQLException{
Timestamp timestamp;
if(value instanceof Timestamp){
timestamp = (Timestamp) value;
timestamp = (Timestamp) new LocalizedTimestamp();

Calendar gmtCal = Calendar.getInstance(GMT);
//where GMT = TimeZone.getTimeZone("GMT-00:00");

st.setTimestamp(index, timestamp,gmtCal);

The catch is that this code will be called on all of our TIMESTAMP types, since we're overriding the existing Hibernate TIMESTAMP type (as LocalizedTimestamp). However, this could be easily resolved by the generated DAOs setting some flag on the LocalizedTimestamp object. As it is, some of us are of the mindset that all the timestamps we store should be GMT, so this still works.

The alternative, producing the same result, is as follows:
/usr/local/mysql/bin/mysqld_safe --user=root --default-time-zone=UTC &


st.setTimestamp(index, timestamp);

Monday, November 07, 2005

mysql 5 and localized dates

Problems with persisting timestamps in Mysql 5 via JDBC:

Cannot convert value '2004-08-16 12:00:00' from column 4(2004-08-16 12:00:00 ) to TIMESTAMP.

java.sql.SQLException: Invalid value for getInt() - '^A' in column 7

3.2.0 alpha
Caused by: java.sql.SQLException: Unknown type '16 in column 6 of 21 in binary-encoded result set.

with the following code:

st.setTimestamp(index, timestamp,gmtCal);

where st is a PreparedStatement and the gmtCal is a Calendar instance set at GMT.

Also, the URL connection string was:

Tuesday, October 25, 2005

Bike fitting

This was fantastic:

Tuesday, October 18, 2005

UTF-8, Java, XML and my linux kernel

Posts to Dom4j after receiving "?" instead of unicode chars while parsing XML.
I've searched and searched for my problem, and this is my last hope.Apache Maven 1.x uses Dom4j for various xml processing and provides a Jelly tag for parsing XML into a DefaultDocument. I'm experiencing Unicode characters being translated into "?" and the hooks are definitely not what I need.

I've replaced the 1.4 version shipped with Maven to 1.6.1 and have
created a simple test goal in order to replicate the behavior. I've
tried to two different ways of parsing the file, one through using a StringReader that's opened a file in UTF-8, and the other using the default xml:parse tag in Jelly. Both with the same results. So, my question is, has anyone else experienced anything similar? Should I be posting my question on the Maven list? Any ideas?

Thanks in advance,

Pretty much after I sent my post I realized that this could be a
JVM/System issue. Further testing, and loading NLS into my kernel
resolved the problem. The System.out issue was resolved by setting my LC_ALL="en_US.utf8" and then I proceeded with my kernel modification.

Friday, October 14, 2005

Ruby FileFinder now toast, gse-locate.el is in

I was really getting tired of my little Ruby file finder app due to the cost in performance. If anything else was running it became sluggish and was slow to start. I intended to write a replacement in C using a database, probably SQLite. I was already making plans and designing what I'd need to do. Then Norm suggested I use slocate. Conveniently, there are two Emacs lisp chunks that are already doing that! One is fff.el and the other is gse-locate.

First thing is to create a "locate" database. I wanted to index everything in /ext, so the command was "slocate -U /ext -o ext.files.db". I modified gse-locate to work in linux and the final code is:

;; gse-locate.el
;; Summary: Emacs front-end to 'locate'.
;; Author: Scott Evans
;; Home:
;; Time-stamp: <2004.12.22 23:20:59 gse>
;; Commentary:
;; A handy Emacs interface to 'locate'. Having this interface
;; around is great -- it gives you a one-step way to get around a
;; filesystem, and spend a lot less time navigating directory trees.
;; I lifted a lot of the major mode code from Steve Molitor's
;; rec-files.el (or at least from my modified version).
;; I spend most of my time in Windows now, and porting slocate to
;; Cygwin didn't go real well so I wrote my own simple 'locate'
;; replacement for Cygwin (see
;; Written for/using XEmacs on Windows. Changes are welcome.
;; Installation:
;; (require 'gse-locate)
;; and if you like:
;; (global-set-key '[f10] 'gse-locate)
;; Change Log
;; ----------
;; 2005.10.11 Modified to use locate and work in Linux -RR
;; 2004.01.18 Actually, -c is bad.
;; 2004.01.18 Use loc -c, take out backslash replacing.
;; 2004.01.18 Add gse-locate-execute-file.
;; 2004.01.18 Change to major mode with keybindings. Add some customization.
;; 2002.04.08 Created.

;; Unix (and native Cygwin) users can probably just set this
;; to "locate".
(defvar gse-locate-command "locate"
"*\"locate\" shell command. This will be used as an argument to
shell-command, with the search pattern concatenated to it.

This specified program should take a string (or regular expression, if
you intend to use them) as a parameter, and return a list of files to
stdout, one per line.")

(defvar gse-locate-hooks nil
"List of functions to call when entering gse-locate-mode")

(defvar gse-locate-regexp-list
(list "\\.elc"
"A list of regular expressions that match \"uninteresting\" filenames.
These will be stripped from the locate list before it is displayed.")


(defvar gse-locate-prev-wconfig nil)
(defvar gse-locate-buf nil)
(defvar gse-locate-history nil)


(defun gse-locate (pattern)
"Lightweight interface to locate. PATTERN is the string or regexp
that will be passed to locate (see gse-locate-command).

If exactly one file matches, it will be opened. Otherwise a list
of files will be presented (see gse-locate-mode)."
(read-from-minibuffer "locate pattern: " nil nil nil

(set-buffer (get-buffer-create "*locate*"))
(setq buffer-read-only nil)
(shell-command (concat gse-locate-command " -d /ext/ext.files.db " pattern) t)

;; Clean up stuff we don't want to see.
(let* ((i 0))
(while (< i (length gse-locate-regexp-list))
(let ((cur-regexp (nth i gse-locate-regexp-list)))
(goto-char (point-min))
(delete-matching-lines cur-regexp))
(setq i (+ i 1))))

(let ((number-matches (count-lines (point-min) (point-max))))
((= number-matches 0)
(message "No matches."))
((= number-matches 1)
;; One match. Open the file.
(find-file (buffer-substring
(goto-char (point-min))
;; Multiple matches.
(setq gse-locate-prev-wconfig (current-window-configuration))
(goto-char (point-min))
(switch-to-buffer (current-buffer))
(setq gse-locate-buf (current-buffer))


(defvar gse-locate-mode-map nil
"Keymap for gse-locate-mode.")

(if gse-locate-mode-map
(setq gse-locate-mode-map (make-sparse-keymap))
(define-key gse-locate-mode-map "v" 'gse-locate-select-this-window)
(define-key gse-locate-mode-map "\C-m" 'gse-locate-select-this-window)
(define-key gse-locate-mode-map "o" 'gse-locate-select-other-window)
(define-key gse-locate-mode-map " " 'next-line)
(define-key gse-locate-mode-map "n" 'next-line)
(define-key gse-locate-mode-map "p" 'previous-line)
(define-key gse-locate-mode-map "q" 'gse-locate-quit)
(define-key gse-locate-mode-map "1" 'gse-locate-select-1-window)
(define-key gse-locate-mode-map "?" 'describe-mode)

(when (functionp 'mswindows-shell-execute)
(define-key gse-locate-mode-map "X" 'gse-locate-execute-file))


(defun gse-locate-mode ()
"Lightweight major mode to select a file from \"locate\" output.

Special keys:
(setq major-mode 'gse-locate-mode)
(setq mode-name "Locate")
(use-local-map gse-locate-mode-map)
(setq truncate-lines t)
(setq buffer-read-only t)
(run-hooks 'gse-locate-hooks))


(defun gse-locate-current-file ()
(search-forward-regexp "$"))))


(defun gse-locate-find-file (file-name &optional find-function)
"Open file if it exists."
(when (not find-function)
(setq find-function 'find-file))

(if (file-exists-p file-name)
(funcall find-function file-name)
(error "%s%s%s" "File '" file-name "' does not exist!")))


(defun gse-locate-select-this-window ()
"Select this line's file in this window."
(gse-locate-find-file (gse-locate-current-file)))


(defun gse-locate-select-other-window ()
"Select this line's file in this window."
(gse-locate-find-file (gse-locate-current-file) 'find-file-other-window))


(defun gse-locate-select-1-window ()
"Select this line's buffer, alone, in full frame."
(gse-locate-find-file (gse-locate-current-file))
(bury-buffer (other-buffer))


(defun gse-locate-quit ()
"Close gse-locate buffer."
(let ((buf (get-buffer gse-locate-buf)))
(and buf (bury-buffer buf)))
(set-window-configuration gse-locate-prev-wconfig))


(defun gse-locate-execute-file ()
"Launch this line's file. Currently windows-only (and probably

(let ((file-name (gse-locate-current-file)))
(if (file-exists-p file-name)
(mswindows-shell-execute nil file-name)
(error "%s%s%s" "File '" file-name "' does not exist!"))))


(provide 'gse-locate)

Monday, October 10, 2005

jde-usages, this is great

Quite happy to have run across the jde-usages plugin for the Java Development Environment for Emacs. The usages plugin works great with the prj.el generated from Maven. The only tweak I've found is adding classpaths of other projects that depend on the one in which you're working. Primarily this is useful for refactoring methods and seeing which classes will be affected. Usages is also quite handy building trees of subclass/implementations. Once the cache is running it's quite speedy.

Friday, October 07, 2005

Beagle 0.1.1 in Gentoo

Following these instructions ( I downloaded app-misc/beagle-0.1.1.ebuild,files/beagle-0.1.1-null-task-source.patch,files/beagle-0.1.1-uri-serialization.patch and placed them in /usr/portage/app-misc/beagle. Actually, the .patch files go in the beagle/files directory.
Then "ebuild /usr/portage/app-misc/beagle/beagle-0.1.1.ebuild digest".

Lastly, my /etc/portage/package.unmask:

and /etc/portage/package.keywords:
sys-apps/dbus ~x86
app-misc/beagle ~x86
dev-lang/mono ~x86
dev-dotnet/libgdiplus ~x86
dev-dotnet/gecko-sharp ~x86
dev-dotnet/gtk-sharp ~x86
dev-libs/gmime ~x86
dev-dotnet/gnome-sharp ~x86
dev-dotnet/art-sharp ~x86
dev-dotnet/gnomevfs-sharp ~x86
dev-dotnet/glade-sharp ~x86
dev-dotnet/gconf-sharp ~x86
media-libs/libexif ~x86

Finally, "emerge -av app-misc/beagle" and selecting "yes".

I haven't upgraded my kernel, with the inotify stuff.

Here's my make.conf:
USE="x86 oss apm arts avi berkdb crypt cups encode foomaticdb gdbm gif gpm gtk gtk2 imlib
jpeg kde gnome libg++ libwww mad mikmod motif mpeg ncurses nls oggvorbis
opengl pam pdflib png python qt quicktime readline sdl slang spell ssl
svga tcpd truetype X xml2 xmms xv zlib gtkhtml svg objc
ruby mono dbus eds"

I tried adding "webservices", but it hosed the build.

Tuesday, September 27, 2005

Location equivalency headaches

Bryan and I came up with this while trying solve migration and location/address integrity issues.

To define how multiple Location subordinates are equivalent, consider the following. The Location (country) USA has "states", "commonwealths", and a "district". It's neighbor to the north, Canada, has "provinces" and "territories". Likewise, the UK has "counties" or "post towns". On a formal level, these could all be called, "level 1 subordinates". If you have a Location table that defines a location by its type (and an id), then you could have the following:

id type name
-- ---- ----
1 country USA
2 country Canada
3 country UK
4 level1 Alabama
5 level1 Manitoba
6 level1 Southhampton
7 level1 Puerto Rico
8 level1 District of Columbia
9 country Mexico

id type
-- ----
1 country
2 level1
3 level2
4 level3
5 level4

Note, "level*" indicates a level of subordination. Using a LocationAssociation table you can build a hierarchy of locations and their subordinates.

For address creation, how could you equate like levels and provide an appropriate naming scheme in the context of the country? Here's one possible set of tables providing a solution:

First is a Subordinate Types table that defines the types you will use to identify subordinates (any level). This table could have a translatable counterpart that provided localized identifiers (I18n) per row, thus enabling translations for a single word, in multiple languages. (This is beneficial in the following model because it constrains translation to a single word and the context is derived based on equivalency relation.)

Subordinate Types
id type
-- ----
1 state
2 province
3 territory
4 county
5 district
6 commonwealth

The Subordinate Equivalence table will relate a location, a locationType, specify if the row is a default (more on this in a moment), subordinateType, and a pointer to a localized string used to identify that component in the native country (locationId). This table realizes the equivalent nature of LocationTypes and SubordinateTypes in context of the owning location (usually country).

Subordinate Equivalence
id locationId locationType defaultBit subordinateType defaultLocalizedStringId
-- ---------- ------------ ---------- --------------- ------------------------
1 1 level1 1 state ->"State"
2 1 level1 0 district ->"District"
3 1 level1 0 commonwealth ->"Commonwealth"
4 2 level1 1 province ->"Province"
5 2 level1 0 territory ->"Territory"
6 3 level1 1 county ->"County"
7 3 level1 0 posttown ->"Post Town"
8 1 level2 1 county ->"County"
9 1 level3 1 city ->"City"
10 2 level3 1 city ->"City"
11 3 level4 1 city ->"City"
12 9 level1 1 state ->"Estado"
13 9 level3 1 city ->"Ciudad"

Thus, this table relates that "level3" is common across USA, Canada, UK, and Mexico. But, "level2" has no associated alternatives *in those countries*. Likewise, the word "county" is used twice, once to show a "county" is "level2" locationType in the USA. BUT, it also shows that it is a "level1" for the UK. So, the UK "county" could be considered equivalent to a USA "state" and Canadian "province" and "territory". This is quite helpful to know in address construction and UI presentation. The "defaultBit" shows which is the default subordinate type for that level, to display to the user.

I18n Benefits:

With the use of the FK to the Subordinate Types table, the i18n value of the subordinate type can be easily fetched. This is most helpful for countries that support multiple languages (ie. Canada or Switzerland).

Likewise, if a user in the USA wants to know what an "Estado" is, we can determine the equivalent through this table and provide an appropriate translation.

Monday, September 26, 2005

Lexmark printing
as root:
emerge deb2targz
execute deb2targz on the driver
execute the shell script located in usr/local/lexmark/setup.lexprint

execute lexprint to setup the printer

At this point the CUPS entry is created (once the printer is added) and you can print from anywhere that supports CUPS.

Oracle settings

This is what I had to add to my .bashrc to get Oracle drivers/sqlplus working:
#oracle stuff

Wednesday, September 21, 2005

Opera Adblock. thanks to the wiki

Here's how I finally got "AdBlock" functionality working in Opera.

In my opera6.ini under "[Image Link Popup Menu]" and "[Image Popup Menu]:

Item, "Destroy image python" = Copy image address & Execute program,"/home/russ/.opera/","%c" & Select user CSS file,1 & Deselect user CSS file, 1 & Select user CSS file,1 & Deselect user CSS file, 1

Where (universally executable) is:

import sys, os


# read current CSS

# write new CSS
if len(sys.argv)==2:
elif len(sys.argv)==3:
elif len(sys.argv)==5:
elif len(sys.argv)==4:

if (cssStr=="\n")|(cssStr==""):
cssFile.writelines('\n{display:none !important;}')


And now that Opera is free without the ad banner, imports FireFox bookmarks, and can now block images this may be a long-term relationship. It's fast and so far I like it.

Forgot that this is also present in opera6.ini:

[Author Display Mode]
Author Font and Colors=1
User Font and Colors=0
User Link Settings=1
Author CSS=1
User CSS=1

[User Display Mode]
User CSS=1
User Link Settings=1
Author CSS=1
Author Font and Colors=1
User Font and Colors=0

[Local CSS Files]
Name 12=Adblock
File 12=adblock.css
Name 13=Overlay
File 13=overlay.css
Active 13=0
Active 12=1

Saturday, September 17, 2005

Divs, Iframes, and Windows rendering

Been a long weekend debugging a bunch of pop-ins that we're using that were hosed because of IE's rendering of Selects. The divs that were being popped-in overlapped some selects are were thus subject to the nasty effect of the Selects bleeding through. I was trying to glean code from the DatePicker.js in Tapestry, but the way it automatically generates the coordinates for the calendar was a little too tricky for what we need. And, I couldn't get it to work right (after 2-3 hours!). Thankfully it was helpful enough to illustrate the necessity of generating an IFrame (which takes rendering precedence) under the div. I had that working except in some cases the iframe was out of synch with the div so the locations appear hosed.

Anyhow, long story short, thanks to this blog from Joe King at MS, it clicked that I was 90% of the way there but was missing only a couple small things. After another hour of tweaking, I have a solution.

Assuming you have a DIV that is your pop-in already placed where you want it to show, it's just a matter of adding the following code and calling "hideOrShowPop" (stupid, yes, I should rename it):

function hideOrShowPop(id){
var popin= document.getElementById(id);
var idToFetch = id + "Active";
var popinActive = document.getElementById(idToFetch);


var clone = document.getElementById("clone"+id);
if(clone != null){"hidden";

function handlePop(id){
var popinDiv = document.getElementById(id);
var anchorNode = document.getElementById(id+"Anchor");"visible";"block";

var cloneDiv = popinDiv.cloneNode(false);
//this is for figuring out how to put the popin ANYWHERE on the screen, currently not working
//var anchorPoint = getLocationPoint(anchorNode);
// = anchorPoint.y;
// = anchorPoint.x;

var width = parseInt(;
var height = parseInt(; = "clone" + id;
cloneDiv.innerHTML = "";
} catch(e){
alert("Popin exception caused by id: " + idToFetch + " or " + id);

So the code clones the div, then sets the inner node of the cloned div to an iframe. Lastly it's appended to whomever owns the original div, and then the original div is appended after the clone.

Now I can go to bed.

Friday, September 16, 2005

Firefox HTTP sniffer

Credit to Todd T. for this fantastic find:

Easily view HTTP traffic from your browser AND edit it before it's sent.

Thursday, September 15, 2005

16080x1050, finally!

Modeline "1680x1050" 147.14 1680 1784 1968 2256 1050 1051 1054 1087

Tapestry: DatePIcker, SSL, and IE

Final solution was adding "src=https:Home.html" to the IFrame, since Home.html was relative. This fixed the annoying security issues, and it left the IFrame in place (so no Selects will ever be hosed when a DatePicker is near). What a dang pain.

Here are the posts I wrote along the way (most recent first):

Branding myself a hasty hoser, the previously posted code did not work (my test page was cached with a previous attempt).

So, for real this time, the fix is: commenting out that line of code for generating the IFrame. Of course, if you have a dropdown underneath it then you'll probably want it, but that's a tradeoff explicitly awaiting IE developers. This fix is quick and dirty, but it works for me since we don't have any Selects near DatePickers.;en-us;177378
" However, IFRAME tries to draw on top of SELECT because it is last in the order. To prevent this, the value of the SELECT element's z-index must be greater than that of the IFRAME, or the SELECT element must be last in the list to ensure that the element always draws on top of the IFRAME when the z-indexes are the same. Because of this, z-indexing similar content can be difficult."

IFrame security covered here (Dec 04, which explains why the DatePicker was working fine in IE last year at this time):

Hopefully I'm done with this topic, apologies for the repeated posts,

Thanks to this thread,, I have changed DatePicker.js:

(previous line)
underDiv.innerHTML = "";

(new code)
underDiv.innerHTML = "";

Works wonderfully.

On 9/15/05, RR wrote:

>opening the DatePicker window yields a security warning in Internet
>Explorer when using an SSL connection. The message is the infamous "This
>page contains both

secure and nonsecure items. Do you want to display
>the non

secure items?".

In Tapestry 3 I'm having the same problem as above -- wondering if anyone else has experienced it and found a solution? This thread is old and I haven't found anything else related.

What I've tried (and without resolution):

1)added my cert as a trusted cert
2)changed the component render to place an onclick in the image instead of wrapping it as an anchor
3)explicitly added "https:" in front of the rendered src attribute on the img.

Any other ideas out there?

Java TCP Tunnel analyzer

Very slick little GUI in soap-2.3.jar ( for setting up a proxy and monitoring TCP traffic. Basic usage is:

java -cp soap-2.3.jar 9999 localhost 8080

Where "9999" is the local socket, localhost is the destination host to monitor, "8080" is the port to monitor on the destination host. So, to monitor your communication with Gmail you could execute: java -cp soap-2.3.jar 9999 80. Then, in your browser, you'd use the following URL:


However, this will only work if you don't require an HTTP proxy.

Likewise with https with a local secure link and server:
java -cp soap-2.3.jar 8888 localhost 8443

Wednesday, September 14, 2005

3550 entry

Educational goals:
a) Finish my CS degree before I'm 30. Hopefully going to happen in the time frame I'd like. Work, kids, and other responsibilities have outweighed my educational pursuits since I began working full-time in 2000.

b) Obtain a Masters degree in something. While I'd like to think that I'd focus on CS studies once I'm done at UVSC, my wife is pretty sure that we'll be taking a break from school for a while. It'd be all too convient if UVSC offered a Masters program by the time I'm done with my under-grad.

Some interests:

  • Linux

  • Programming

  • Being with my kids

  • Software Development

  • Soccer

  • U.S. History

  • Reading (mostly non-fiction right now)

Friday, September 09, 2005

Tapestry render solution

The solution to my render rant a couple days back has been tedious, but not difficult. Started out with:

* Simple interface declaring required properties for a protected component to (en/dis)abling a component render and its disable(d) state.
public interface IProtectedComponent{

* Flag for allowing component render.
* @return a Boolean value
public boolean getAllowRender();


Then I wrapped all of the default Tapestry components we're using, for example:

* A protected PropertySelection component only rendered if allowedRender is true.
public abstract class ProtectedPropertySelection extends PropertySelection implements IProtectedComponent{

protected void renderComponent(IMarkupWriter writer, IRequestCycle cycle){
if(getAllowRender() == true){
super.renderComponent(writer, cycle);

Did this for:

Then, in the base class pageBeginRender, added the following:

for(Iterator componentIter = components.keySet().iterator(); componentIter.hasNext();){
String componentId = (String);
IComponent component = getComponent(componentId);
IBinding elementId = component.getBinding(ELEMENT_IDENTIFIER);

IProtectedComponent protectedComponent = (IProtectedComponent)component;
if(elementId != null){
//resolve rights to component
//boolean updateOrExecute = canAccess(elementId, CRUDAction.UPDATE);
boolean updateOrExecute = true;
if(! updateOrExecute){
//boolean canRead = canAccess(elementId, CRUDAction.READ);
boolean canRead = false;
//do not render the component
component.setProperty(ELEMENT_ALLOW_RENDER, "false");
component.setProperty(ELEMENT_DISABLE, Boolean.TRUE);
//Render the component, but make it disabled
component.setProperty(ELEMENT_ALLOW_RENDER, "true");
component.setProperty(ELEMENT_DISABLE, Boolean.TRUE);
//Render the component normally
component.setProperty(ELEMENT_ALLOW_RENDER, Boolean.TRUE);
component.setProperty(ELEMENT_DISABLE, Boolean.FALSE);


So, I'm basically cycling through all of the components and changing state based upon id attribute and rights to the element. By the time renderComponent is called, the allowRender and disabled properties are set. Thus, this ensures the component is handled properly based upon user rights associated to the id.

Wednesday, September 07, 2005

Tapestry rant

Okay, time to spout off about Tapestry. Situation is this: let's say you have a page with a bunch of components. And this page happens to contain *secure* elements. Requirements state that the component can be viewable/read-only, updateable, or not present. Security model dictates, some time before render, what the state of each component should be. So, next issue is rendering the components. Should be as easy as updating the "disabled" attribute for the read-only, and eliminating/removing/not-rendering the components that are unauthorized.

What are the reasons for Tapestry completely hosing the easibility of this process?! This would be cake in XMLC or Wicket!

1. Components must be defined prior to render. Which means, you need to have the "disabled" parameter set to false on *every* single component you want to protect, at build-time. It is a pain to add a "binding" on at run-time and highly discouraged (

2. Tapestry is incapable of removing components at render time. You don't have access to the DOM, you can't set a flag, nothing. I should clarify, this is the case for a) all default Tapestry components b) you can sort of work around this by wrapping the component in a Conditional component.

3. Tapestry default components do not allow render flags. In conjunction with issue #2, you can't just set a flag informing Tapestry to NOT render the component.

So, why does this really tick me off? Because I have many pages on which I now need to implement security (aforementioned implementation). I cannot, out-of-the-box, do what I need. I must go through each component and add the "disabled" attribute (only 130+ of them). I must choose to wrap the components in Conditionals if I don't want them rendered. IMO, this is a ton of work and really defeats the purpose of dynamic page generation.

My acceptable is solution is to go through and create wrapper components for each of the Tapestry components we're using. I'll add a default "disabled" attribute that I can hose with at run-time (so the user doesn't have to add it specifically) and a flag that determines whether the component should render. This is going to take more time than I likely have. Wish I knew about this issues months ago, I *may* have reconsidered my choice of web application framework.

One alternative, Java Server Faces, does exactly what I want in its UIComponentBase:

In conclusion, the more I work with Tapestry the more I feel like my hand is being held and while it provides some nifty aspects (reusable components, form validation), I can accomplish essentially the same thing in some other frameworks. The more I look into JSF, the more I'm realizing its value....

Wednesday, August 24, 2005

IceWM whips Flux

Pros (so far):

Smart window placement
faster startup/smaller footprint
ease of customization, configuration files much simpler
good preference tools
invokable menu by key-binding
gaim doesn't crash (compared to flux 0.9.12)
mouse focus more consistent--this drove me nuts in Flux

Tomcat whips JBoss

In order to deploy CDOL into Tomcat 5.x:

1. download Tomcat 5.x, or

2. [un] tar/zip the Tomcat download.

3. modify this snippet to correctly point to the right properties file:




4. modify "//server.xml" file by adding the above snippet, insert it after line 378 (following and before ).

5. modify "//web.xml" by replacing the "" element with:
maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
enableLookups="false" disableUploadTimeout="true"
acceptCount="100" debug="0" scheme="https" secure="true"
keystoreFile="${user.home}/.keystore" keystorPass="yourpassword"
clientAuth="false" sslProtocol="SSL" />

6. copy "~/.maven/repository/commons-logging/jar/commons-logging-1.0.4.jar" into "//common/lib"

7. copy "~/.maven/repository/log4j/jars/1og4j-1.2.8.jar" into "//common/lib"

8. copy "~/.maven/repository/j2ee/jars/j2ee-1.4.jar" into "//common/lib"

9. lastly, to run the build/deploy to build the war and copy into "//webapps"

10. startup tomcat, "//bin/ run"

Using Tomcat has sped up my build/deploy time from >=160s to <=30s, hooray for Tomcat.

If you're on Linux and downloaded the JDK from Sun, double-check that "javac" has executable permissions.

Tuesday, August 23, 2005

Localized dates the JDBC way

An investigative coworker (Todd) found this that explains a method of translating a given Calendar object into a GMT Timestamp. Likewise, retrieving the date in a specific locale can be achieved through ResultSet.getTimestamp the Timestamp can be localized. Now that all of the code for *doing* this is already in place it'd be a mighty pain to rip it out and refactor.

Implementing the above is a great option if you need a blanket solution. Frankly, I agree with Todd that we should be storing everything in GMT anyway. At this point in the game that's not feasible and we need specific GMT/Localizable functionality on only a few specific columns. Future situations may present alternative necessities.

Friday, August 19, 2005

Maven and Emacs, the romance and beauty

Hot dog, this is great:
Maven JDEE plugin
more functionality here
and here.
That last one is especially great because of the ability to run goals with the Maven console. Really speeds up dev time having it open and instantly executing goals--with the ability to click on compile errors and have them immediately opened and focused in the primary Emacs frame. Loving it so far.

Monday, August 15, 2005

copy to /usr/portage/distfiles
emerge cisco-vpnclient-3des

copy .pcf to /etc/opt/cisco-vpnclient/Profiles

vpnclient connect <profile> user <username>

because Windows keeps jacking my clock

Of course the real solution is to eliminate the Windows partition altogether--and if I was done with school, that'd be acceptable.

Instead, here's a Ruby time synchronization hack/work-around I'll place in my startup script.

require 'net/http'
require 'date'

#handle the case when the proxy_addr is null

while(date == nil)
#fetch date
Net::HTTP::Proxy(proxy_addr,proxy_port).start('') {|http|
http.read_timeout = 1
http.open_timeout = 1
p = http.request_head('/')
puts p['content-type']
puts p['date']
date = p['date']
puts "is net connected?"

#set date
if(date != nil)
puts date
datetime = DateTime.parse(date)
puts datetime
mst = (datetime.new_offset(-0.25) )
#will need to determine if this is during daylight savings or not and adjust fraction accordingly
puts mst

#parse and format for posix
dates = mst.to_s.split('-')
times = dates.to_s.split(":")
finalDate = dates[1]
finalDate += dates[2][0,2]
finalDate += times[0][9,11]
finalDate += times[1]
finalDate += dates[0]
puts finalDate
`date #{finalDate}`

Friday, August 12, 2005

Localized date resolution

Changing the datatype on our Oracle columns to Timestamp was one important step. Next, LocalizedTimestamp was created (implementing Hibernate's UserType for necessary integration) extending Timestamp and adding a bunch of constructors for multi-purpose instantiation. Moved all of the date converstion stuff into a separate abstract utility class. Followed the code-generation logic in the previous post and added a separate getDateValueLocalized(TimeZone) for specifically localized date columns (also, the date is GMT-ized going in on those same columns).

Took about two days, with the bulk of the work being in the refactoring of existing code and ensuring correct behavior via unit tests.

Wednesday, August 10, 2005

more headaches with localizing dates

Per the previous post we have decided upon normalizing dates into standard UTC/GMT times (at midnight of the day they occurred). Using the code generation tools that we have it was simply a matter of adding the necessary logic in the setters and getters of those fields on the domain model objects. So, if a date field is a GMT date we add in:

this._dateValue = new java.sql.Timestamp(setDateToUtcHourFromTimeZone((java.util.Date)dateValue).getTime());

Where setDateToUtcHourFromTimeZone is going to find the TimeZone in the session, modify the date value to midnight at that TimeZone and then translate the midnight date value to real UTC/GMT time.

Likewise fo the getter:

return new java.sql.Timestamp(getTimeZoneDate((java.util.Date)_dateValue).getTime());

Where getTimeZoneDate will translate the Date (which is GMT) back into the correct TimeZone date.

So, what's the problem with this? The problem is Hibernate calls the getters and setters. It's just fine and dandy when you set the value via the object. But, once that DMO gets into Hibernate's clutches it calls the getter on it which effectively negates the intended GMT normalization! And again, when Hibernate creates the object and calls the setter the GMT date retrieved from the database is then retranslated into a different value rather than the desired the translated local TimeZone date ).

What's the solution? If you're using Oracle , evaluate the Timestamp with timezone datatype. Same thing with PostgreSql. If not, what now? Use a varchar and muck with the values through the data access code? That's nasty and then you lose all major benefits of the RDBS datatype operations. Varchar operations could be more CPU intensive compared to numerical datatypes, especially as record sizes ramp. If I find out anything more I'll update.


So it's not going to be too difficult to fix this issue. The changes will be as follows:
1) generated DMO code will not hose with the getter, it will return the value as-is.

2) generated DMO code will have additional method for obtaining the local date value given a TimeZone.

3) generated DMO code will still use the above function but its signature will change to use a LocalizedTimestamp object.

4) LocalizedTimestamp object to be created. Has a Timestamp value and a TimeZone value. Default TimeZone is GMT.

Implementation details

The setter calls setDateToUtcHourFromTimeZone and passes in the LocalizedTimestamp. Internal logic evaluates that this is either a GMT Timestamp or not. If it is, translate, if not, keep original value. This way the code is smart enough to not hose the value if it is already GMT.

As for the getter, we'll toss on a new method (getLocalDateValue) that will require a TimeZone and will translate the value for the caller.

Thus, we satisfy the setter/getter issue for Hibernate operations and still provide necessary functionality for the additional callers.

Wednesday, August 03, 2005

test #4, w/4.6 dev

ran again with -Xmx1200m with same results.
java -Xmx1024m -classpath /home/russ/.maven/repository/junit/jars/junit-3.8.1.jar:/home/russ/.maven/repository/ims/jars/imsobjects-1.2.0-SNAPSHOT.jar:/home/russ/.maven/repository/db4o/jar
s/db4o-4.6-java1.2.jar:./target/test-classes junit.textui.TestRunner

.Loading database with: 1000000 ids(2x), names, people

573565ms elapsed for task
Looking up Person by id 113322
1ms elapsed for task
Looking up Name from Person
[db4o 4.6.004 2005-08-03 22:47:42]
Uncaught Exception. Engine closed.
[db4o 4.6.004 2005-08-03 22:47:43]
Please mail the following to


Closing database
0ms elapsed for task

Friday, July 29, 2005

test #3

java -Xmx1024m -classpath /home/russ/.maven/repository/junit/jars/junit-3.8.1.jar:/home/russ//.maven/repository/ims/
jars/imsobjects-1.2.0-SNAPSHOT.jar:/home/russ/.maven/repository/db4o/jars/db4o-4.5-java1.4.jar:. junit.textui.TestRunner
.Loading database with: 1000000 ids(2x), names, people

502640ms elapsed for task
Looking up Person by id 113322
1ms elapsed for task
Looking up Name from Person
[db4o 4.5.009 2005-07-29 10:48:45]
Uncaught Exception. Engine closed.
[db4o 4.5.009 2005-07-29 10:48:45]
Please mail the following to


Closing database
39ms elapsed for task
Time: 569.755
There was 1 error:
1) testLookupByRfn( Uncaught Exception. db4o engine closed.
at com.db4o.YapStream.fatalException(Unknown Source)
at com.db4o.YapStream.get1(Unknown Source)
at com.db4o.YapStream.get(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
at sun.reflect.DelegatingMethodAccessorImpl.invoke(

Tests run: 1, Failures: 0, Errors: 1

public void setUp(){

database = Db4o.openFile(DB_FILENAME);

---loading code---
IMSID ims = new IMSID("" + (peopleCntr * 17));

TBasicPerson newDude = new TBasicPerson();

TId imsId = new TId();


TId rfn = new TId();
rfn.setIdValue("" + (peopleCntr * 31));


TName name = new TName();
name.setNameValue("Some Test dude name" + (peopleCntr * 19));

---end loading code---

TId queryId = new TId();

Query query = database.query();

ObjectSet results = database.get(queryId);

beginLoggedEvent("Looking up Person by id " + queryId.getIdValue());
assertTrue(results.size() == 1);
TBasicPerson person = ((TId);

beginLoggedEvent("Looking up Name from Person");
TName queryName = new TName();
results = database.get(queryName);
assertTrue(results.size() == 1);
System.out.println("Name is: " + ((TName);
---end tests---

Thursday, July 28, 2005

db4o second test

Using a BasicPerson, TestId and TestName pojos:
Loading database with: 1000000 ids(2x), names, people
just reached: 0 in 0
just reached: 10000 in 5235
just reached: 20000 in 9609
just reached: 30000 in 14014
just reached: 40000 in 18555
just reached: 50000 in 24294
just reached: 60000 in 29458
just reached: 70000 in 34711
just reached: 80000 in 38898
just reached: 90000 in 43103
just reached: 100000 in 48126
just reached: 110000 in 52073
just reached: 120000 in 56077
just reached: 130000 in 60128
just reached: 140000 in 64137
just reached: 150000 in 69626
just reached: 160000 in 73657
just reached: 170000 in 77673
just reached: 180000 in 81744
just reached: 190000 in 85833
just reached: 200000 in 89924
just reached: 210000 in 93986
just reached: 220000 in 98120
just reached: 230000 in 102249
just reached: 240000 in 108154
just reached: 250000 in 112670
just reached: 260000 in 117505
just reached: 270000 in 122052
just reached: 280000 in 126539
just reached: 290000 in 131076
just reached: 300000 in 136139
just reached: 320000 in 146089
just reached: 330000 in 150771
just reached: 340000 in 155347
just reached: 350000 in 159983
just reached: 360000 in 164586
just reached: 370000 in 169100
just reached: 380000 in 176931
just reached: 390000 in 181569
just reached: 400000 in 186232
just reached: 410000 in 190950
just reached: 420000 in 195867
just reached: 430000 in 200507
just reached: 440000 in 205159
just reached: 450000 in 209983
just reached: 460000 in 214668
just reached: 470000 in 219328
just reached: 480000 in 224170
just reached: 490000 in 228886
just reached: 500000 in 233544
just reached: 510000 in 238384
just reached: 520000 in 243180
just reached: 530000 in 247650
just reached: 540000 in 252316
just reached: 550000 in 256820
just reached: 560000 in 261308
just reached: 570000 in 266162
just reached: 580000 in 270585
just reached: 590000 in 275104
just reached: 600000 in 279884
just reached: 610000 in 289737
just reached: 620000 in 294251
just reached: 630000 in 299066
just reached: 640000 in 303539
just reached: 650000 in 308344
just reached: 660000 in 312893
just reached: 670000 in 317665
just reached: 680000 in 322080
just reached: 690000 in 326708
just reached: 700000 in 331115
just reached: 710000 in 335714
just reached: 720000 in 340175
just reached: 730000 in 344840
just reached: 740000 in 349324
just reached: 750000 in 353799
just reached: 760000 in 358612
just reached: 770000 in 363367
just reached: 780000 in 367962
just reached: 790000 in 372655
just reached: 800000 in 377622
just reached: 810000 in 382321
just reached: 820000 in 387158
just reached: 830000 in 391964
just reached: 840000 in 396887
just reached: 850000 in 401993
just reached: 860000 in 406987
just reached: 870000 in 411791
just reached: 880000 in 416510
just reached: 890000 in 421543
just reached: 900000 in 426205
just reached: 910000 in 431058
just reached: 920000 in 435586
just reached: 930000 in 440384
just reached: 940000 in 455117
just reached: 950000 in 459988
just reached: 960000 in 464647
just reached: 970000 in 469666
just reached: 980000 in 474506
just reached: 990000 in 486817
499153ms elapsed for task
Closing database
Total objects added: 2mil Ids + 1mil names + 1mil persons = 4mil objects

Next to test lookup on FKs.

Friday, July 22, 2005

db4o intial testing

Thanks to Carl (and a bit of more reading of the tutorial) I updated my test to index on a single field--and then queried on that field. New results:

Million records took: 128966 ms
Fetching a record from a million: 5 ms
Deleting a million: 132258 ms

*Very* impressive. Perhaps I'll get more creative with a larger POJO and evaluate further metrics. The numbers for creating and deleting records are larger due to more processes running at the time of the test.

Dell D800 1.7 PM, 2G ram, JVM, from maven using JUnit

Dumped in a million objects (4 field POJO) and the db file grew to 64MB.

Million records took: 48195 ms
Fetching a record (QBE) from a million: 12460 ms
Deleting a million, but result set from (by Query): 48234 ms

Thursday, July 21, 2005

Importance of UTC date normalization in event triggers.

One important aspect of internationalizing an application may be that of date handling. That is, how will the application read and write dates. Should the scope of the project be limited to viewing only, then the point is moot. However, if the need arises that these dates drive some activity or event then "normalizing" the dates should be heavily considered. Choosing an international standard (UTC) and conforming date entries allows date-specific operations and triggers to be precisely executed.

Consider the following example. An organization has dates that drive specific events or triggers. A monitor/daemon is run that periodically evaluates these dates and fires the appropriate action. Presume the monitor awakes at midnight MST. Then the scenario in which an organization in New Zealand depends on the date of 24 July to trigger some action will then be delayed for 33 hours--and the people/systems dependent upon those date-driven operations will be hosed. Here's how:

    00:00 Denver July 23 the monitor awakes. Monitor evaluates all dates
    for 23 July and invokes appropriate operations.
    However, the New Zealand organization will not be processed yet,
    awaiting for midnight Denver on July 24.
    BUT, 00:00 Denver July 23 is 19:00 NZ July 24. The time difference
    is 19 hours. So, the New Zealand organization must wait until 00:00
    Denver July 24. So, 19h + 24h = 43h before the effective date
    processing occurs.

Alternatively, if the effective date is stored as a timestamp (even though only the "date" is really relevant considering the requirement to run operations based on "that day") and the monitor runs hourly the problem is solved. The New Zealand "date" of 00:00 July 24 is really 12:00 July 23 UTC. If the monitor then only evaluates per-hour dates, the application can then address international needs required by the effective date handling. Thus, all triggers and date dependent operations will only ever be delayed by an hour and ensuring that at midnight for any given timezone the correct operations for that timezone's day will be executed.

Thursday, July 07, 2005

more file procecssing stats

So I can't get over the fact that I want to write the GUI in Swing. I had the idea to use Ruby on the back end to get the list of files, then write the whole list to a file. Then use Java to pickup the file, parse it, search (per user input) for the file and use Swing as the GUI! All, of course, within a Ruby script. Sound impossible? Check it out:

time ruby finderPrinter.rb

real 0m0.598s
user 0m0.427s
sys 0m0.164s

So that's 17370 files, written to disk in .598 seconds! Beautiful! File is ~1.5 meg. How about how fast Java can process the file?

java -classpath . JReader
took 169 milliseconds
Looked for 'a' in all files: 17258 169 milliseconds

Incredible. Here's the java code:

public void readFile(String fileName) throws FileNotFoundException{
//see how long it takes to read in the file
//iterate the list
//then search through the list for a given name
long timeStarted = System.currentTimeMillis();
BufferedReader reader = new BufferedReader(new FileReader(fileName));
String currentLine = null;
currentLine = reader.readLine();
catch(IOException ex){

for(Iterator fileIter = fileList.iterator();fileIter.hasNext();){
String filePath = (String);
if(filePath.indexOf("a") > 0){
System.out.println("took " + ((System.currentTimeMillis() - timeStarted)) + " milliseconds");
System.out.println("Looked for 'a' in all files: " + resultList.size() + " " + ((System.currentTimeMillis() - timeStarted)) + "milliseconds");

Pretty simple, but that's exactly what I want. The two Collections (fileList and resultList) are ArrayLists. If it operates this quickly, it *should* be acceptable to get a GUI in place that can execute with these ideas. Real kicker is to run the Ruby script that then kicks off the java code...

Finding files
done writing file, now kicking off java process
took 143 milliseconds
Looked for 'a' in all files: 17260 143milliseconds

real 0m0.895s
user 0m0.630s
sys 0m0.211s

Acceptable? Just have to code the gui to find out.

Wednesday, July 06, 2005

file fetching stats

I ported the ruby code to python and java for the file finder app. I'd love to use Swing to get out of the GTK pain, but the obesity of the JDK is going to hinder that route.

Here are the results:

time python
real 0m0.649s
user 0m0.465s
sys 0m0.182s

time ruby test.rb
real 0m0.598s
user 0m0.383s
sys 0m0.195s
java -classpath . SimpleTester
19946 took 37 seconds

Couple of interesting things to note:

First, the code running each of these tests is identical in nature--it's a simple recursive walk of a directory tree. The Pyton code came up with signifcantly less (~1800) files because it ignores symlinks. Wonder what two files Java didn't account for that Ruby did.

Second, the Java collection used in the test is a HashSet containing the file names (Strings). Changing the implementation to use an ArrayList of Files added a second or two onto the test.

I should write this also in C++ and post results. For now Ruby is the clear winner and my hunch held true that it's the slimmest and fastest of the three.

Test specs:
Dell D800 P4M 1700mhz
2G Ram
Linux rr800 2.6.11-gentoo-r4 #5 Tue May 3 08:32:04 MDT 2005 i686 Intel(R) Pentium(R) M processor 1700MHz GenuineIntel GNU/Linux

Monday, June 27, 2005

Tapestry: opinion on page property specifications

I think it is *bad* practice to define page properties with explicit initial-values. The main qualification for this statement is based upon the premise that the page implements PageBeginRender. I oppose defining explicit initial-values (other than new HashMap(), new HashSet() etc.) because it's unclear to the maintainer how/why certain variables are being set and requires two files open, rather than the one java file, to ascertain page state. In order to be really readable and maintainable the coder should define an initial setup method and then invoke that upon first page load. Thus providing logical class definition (since it *is* part of the object and now resides in its guts) and clear code documentation.

Monday, June 20, 2005

Aaron's cookies

Chips Ahoy White Fudge Chunky cookies are the best of the lot.

Good reasoning for ORM
If you are working with object-orientated programming and relational
databases, you will surely have noticed that these are two different
paradigms. The relational model deals with relations,tuples and sets -
it is very mathematical by nature. The object-orientated paradigm
however deals with objects, their atributes and associations to each
other. As soon as you want to make objects persistent using a
relational database you will notice: There is a rift between these two
paradigms, the so called object-relational gap. A object-relational
mapper (or ORM as a shorthand) will help you bridge that gap.

Well, how does this gap manifest? If you are passing objects around in
your application and sometimes reach the point where you want to
persist them, you will typically open a JDBC connection, create an SQL
statement and copy all your property values over to the
PreparedStatement or into the SQL string you are building. This may be
easy for a small value object - but consider this for an object with
many properties. Thats not the only problem. What about associations?
If your Cat object you want to store has a List of kittens contained?
Do you store them too? Automatically? Manually? What about foreign key

The same applies for loading - let's assume you load a Cat object from
the database and it has a collection of kittens. Do you load the
kittens too? Not load them yet but later? If you load the kittens,
consider each of the kitten object has an association to yet more
objects. In this case, such eager loading may easily load your
complete object tree. Not loading the kittens is however not really
better however - you will need explicit reloading later if you
probably want to access the kittens.

As you can see, the object-relational gap quickly becomes very wide if
you have large object models. And there are a lot more things to
consider like lazy loading, circular references, caching, etc. In
fact, there have been studies that showed that about 35% of an
application code was produced by the mapping between application data
and the datastore.

So what can an ORM do for you? A ORM basically intends to takes most
of that burden of your shoulder. With a good ORM, you have to define
the way you map your classes to tables once - which property maps to
which column, which class to which table, etc

Tuesday, June 14, 2005


Thursday, June 09, 2005

Tapestry vs JSF

"...Erik Hatcher likes to "provoke" JSF-ers, such as David Geary, to
duplicate the Table component. The Table includes links for sorting
and paging through long contents ... these links just work, without
any configuration in the application. This is the Tapestry way -- drop
it in and let it work. JSF's approach is primarily a view component
and apparently can't duplicate this."
"Tapestry has literally taken a project that was stalled in
development hell for almost a year and finished it, rebuilt from the
groun up, in 3 months. We built it from the ground up using Tapestry.
We added things as we learned, but it was all very intuitive. First,
simple forms for customer input and simple screens for reporting."

Known implementations. Interestingly these all have their drawbacks.
Oracle's is an "early access edition", MyFaces is somewhat
proprietary, "The MyFaces version of Tiles Support requires the
MyFaces JSF implementation. It will not work with the Sun RI or any
other JSF implementation." And AjaxFaces looks the most intriguing,
but it's a commercial solution!

Think I need to spend a night hosing around with JSF to evaluate it.

Wednesday, June 08, 2005

FindFile released!

Just finished releasing FindFile-0.1. Happy to be able to finally contribute something to the OSS world. Download is available at RubyForge, I've loved coding this in Ruby and been pleased with how quickly and how great it performs. Gonna be a busy summer, but I hope to knock off some TODOs that I have to enhance it.

Tuesday, May 17, 2005

Getting JDE 2.3.5 working with XEmacs 21.4.5 in Gentoo

(all items done as normal user, assuming emerge privileges)
download cedet 1.0 beta 3
download elib
download jde
emerge app-xemacs/sgml
emerge app-xemacs/os-utils
create ~/.semanticdb
untar cedet, elib and jde in the same directory
from the cedet dir: make EMACS=/usr/bin/xemacs
read INSTALL and add appropriate lines to init.el/customize.el
from jde dir update lisp/makefile to point to the appropriate directories
then make jde
add appropriate load-paths and that's it!
;;;jde stuff here
(add-to-list 'load-path (expand-file-name
(add-to-list 'load-path (expand-file-name
(add-to-list 'load-path (expand-file-name "~/.xemacs/jde/cedet-1.0beta3b/ede"))
(add-to-list 'load-path (expand-file-name
(add-to-list 'load-path (expand-file-name
(add-to-list 'load-path (expand-file-name
(add-to-list 'load-path (expand-file-name "~/.xemacs/jde/jde-2.3.5/lisp"))
(load-file "~/.xemacs/jde/cedet-1.0beta3b/common/cedet.el")

(require 'jde)