Home
enlivend's Journal
 
 [Most Recent Entries] [Calendar View] [Friends]

Below are the 16 most recent journal entries recorded in enlivend's LiveJournal:

    Thursday, May 8th, 2008
    3:53 pm
    JFLI and the memory leaks of doom

    Finally, I've got the OK to open source some of the work I was doing last year. First step is the fixes I made to jfli (Java Foreign Language Interface), now available for CVS download or as a tarball. It's the first time anyone has posted changes to this library in some years, so I took the bull by the horns and gave it version number 0.2.

    For the record, this is what I have changed. It was an interesting object lesson in trying to get two GCs to be nice to each other.

    Memory problems:

    1. add-special-free-action takes a symbol, not a function. If you give it a function it doesn't do anything. In this case, that meant that all global-refs were leaking on the Java side. That's a hell of a pile-up - run anything for long enough and Java will run out of memory.

    2. Lisp processes just accumulated in *process-envs*, which meant that the associated stacks etc end up leaking on _both_ sides of the fence, i.e. into both Java and lisp. My first attempt at solving this involved a call to mp:ensure-process-cleanup but...

    3. Suppose a new thread allocates before mp:*current-process* has been set. Then delete-global-ref might be called, which invokes current-env for the first time on this thread, which goes ensure-process-cleanup with null mp:process. SEGV. Farfetched? Well, it happened.

    4. The "access functions" (calls to anything seen in defvtable, i.e. all the JNI's calls into the JVM) failed to memoize a dereferenced foreign-slot-value which was the same every time, and so burned 56 unnecessary bytes per shot (in lisp). This was reclaimed by the GC but it messed up the allocation figures when I was out hunting for real leaks.

    5. Untimely Finalization

      Consider this little problem which exhibited pathological behaviour from time to time:

      • The special free actions are not run when a GC occurs inside mp:without-interrupts, because that could cause a deadlock if the action function claims any locks (e.g. uses hash-tables). Instead of freeing them, the GC just keeps them alive with their special free actions intact.
      • The system maintains a table of all of the objects marked for special free actions (so the GC can find them all easily). Unfortunately, flag-special-free-action takes O(n2) time for n objects in the table. ["Ouch", says Nick.]
      • This table is enlarged by flag-special-free-action, inside without-interrupts to make it atomic with respect to other calls to flag-special-free-action and finalization.
      • If you're unlucky, all of the GC operations are triggered by the enlargement of this table.

      This final aspect completed a vicious cycle: none of the special objects were ever freed, because all of the GC operations occurred inside without-interrupts and hence their special free actions could not be run at that time. Excessive allocation occured, caused by the enlargement of the table which was always filled again quickly for the same reason. (By "excessive", I mean images bloating to over 1GB in very short order.)

      The recommended solution from lispworks-support was to manually mark-and-sweep generation 0 every 1000 or so allocations. Without it, you'd occasionally get generation 0 trying to climb over 1GB while you're sat there wondering why your emacs was running so slow.

    Non-memory problems:

    1. Exports from JFLI package of box-integer and unbox-integer instead of the documented box-int and unbox-int - I restored the documented behaviour.

    2. I needed more configurable exception handling.

    3. No support for system building - you needed a live JVM connection in order to macroexpand source and so couldn't save the image (well, you could, but when you restarted it you wouldn't be able to connect).

    Fixed by upgrading to LispWorks 5.1:

    1. Occurances of java.lang.NullPointerException, java.lang.ArrayIndexOutOfBoundsException, etc which had no explanation even after reading Sun Java sources.

      This turned out to be caused by a bug in the FLI, which could leave the CPU's direction flag set incorrectly in some cases. When the direction flag was set incorrectly, some optimized memory copying routines would corrupt adjacent objects. The bug, fixed in LispWorks 5.1, affected 32-bit x86 platforms running Linux, FreeBSD or Mac OS X (not Windows).

    Friday, May 2nd, 2008
    11:21 am
    It's so neat
    The Customer wanted to an option to simplify (occasionally) the "run" by dropping one of the tests through which the data has to pass before going on to the next stage. Hmm, I asked, given this is in an innermost loop for almost all the data are you willing to take a mild performance hit so I can check each time around whether to apply this test? We'd rather not, said the Customer.

    This innermost test is a hash-table lookup. The solution is so neat:
    (make-hash-table :test 'true :hash-function (constantly 0))
    and setf that (at the beginning of the run, outside all those loops) on top of the real table.

    Thankyou lisp. (Hey, I get paid to do this stuff).
    Thursday, May 1st, 2008
    4:47 am
    Walter Sawyer died

    For a couple of years after we moved into Cambridge, our neighbour was W. W. Sawyer, mathematician, Professor Emeritus, author of several popular books, and general thorn (for his continued criticism of "modern maths") in the educational establishment's side.

    I heard a couple of days ago that Walter died in February. He was 96. He had been living with his daughter in Toronto and them more recently moved to a nursing home there. I got reports from time to time. He was still sharp, still playing chess.

    Searching for him on the web today, there's less than I would have expected about the man himself, given the influence he had on people I've mentioned him to. There's a good biography here along with a number of his papers; one of his books (Prelude to Mathematics) is currently available online.

    Tuesday, April 22nd, 2008
    11:34 am
    P4COM / LWW binding, release 0.2
    This binding allows access from the LispWorks GUI to Robert Cowham's p4com binding to the Perforce SCM. It's been almost 5 years since I released 0.1 so I reckoned it was time for an upgrade.

    New at 0.2:

    • Improved presentation for "p4 opened"
    • Add "p4 revert"
    • Process P4CONFIG file (so you can work with multiple repositories)
    • Improved albeit incomplete support for filenames containing spaces
    • Incorporates latest version of p4com.dll
    I've been using this daily for years. If I have any other users out there I'll be pleased to hear from them as it might improve the chances of there ever being a 0.3
    Friday, January 4th, 2008
    10:42 am
    European Common Lisp Meeting, Amsterdam, April 19/20
    Arthur Lemmens and Edi Weitz are proud to announce (it says here, and so they should be) the European Common Lisp Meeting 2008.  The meeting will consist of a Sunday full of talks on April 20, 2008, with optional dinners on Saturday and Sunday evening.

    For a list of speakers and more information visit http://weitz.de/eclm2008/
    Thursday, November 22nd, 2007
    11:36 am
    Curly Logo
    It's been a while since I played with Logo. Back in the mid 80's a friend had it on his Amiga; years later and with mixed results I tried teaching it at my kids' school on a bunch of battered old Archimedes. But now, thanks to code monk,  it comes wrapped up in a couple of Javascript files and you can run it in your browser.

    The blurb says:

    In order to see it working, I recommend you use Firefox (2.0) or Safari 3. Curly Logo's turtle should be visible in the upper half the window, and a text area should be visible in the lower half. At the bottom of the text area is a space for you to type Logo commands. If you haven't tried Logo before you could try typing some of the following:

    fd 100 rt 90
    Moves the turtle forward and turns it.
    repeat 3 [ fd 100 rt 90 ]
    Completes the square started by the previous command.
    cs repeat 7 [ setpw 2 * repcount setpc repcount fd 50 rt 360 / 7 ]
    A septagon using the 7 colours of the rainbow and a varying pen width

    In order to use Curly Logo you'll need a browser that implements XML, XHTML, SVG, CSS, and JavaScript (Totally Buzzword Compliant). In 2007 that means you need Firefox 2 or Safari 3. Internet Explorer support is not. You might be able to convince me on esoterica like Opera (it kind of half works in Opera without me having tried at all).

    You do not require an internet connexion; once you have downloaded the Curly Logo page and associated JavaScript pages, Curly Logo does not use the network. That means you can save the page and reload it later from your local disk. Curly Logo is not a AJAX application (it is a JAX application), it does not evaluate anything by sending it to a server and expecting a response; all computation and display is performed locally in the web browser by executing JavaScript.

    OK, it's not lisp (but it got me thinking - what's the pain in embedding a smallish lisp in Javascript?). But I have a certain fondness for it.


    Wednesday, October 17th, 2007
    10:47 am
    lisp on JVM
    Rich Hickey posts the following to the jfli-users mailing list::

    As someone interested in Foil or jFli, I thought you might want to 
    know about my latest project - Clojure, a dialect of Lisp for the 
    JVM. It's currently alpha, but fairly complete. I'm looking for some 
    feedback from some intrepid folks willing to kick the tires.

    http://clojure.sourceforge.net/

    Please use the Google group for feedback:

    http://groups.google.com/group/clojure
    Wednesday, August 15th, 2007
    9:54 am
    ILC 2007 accounts

    A summary of the accounts of International Lisp Conference Limited will be found at


    http://www.alu.org/admin/ilc07-accounts.txt
    Thursday, July 19th, 2007
    4:47 pm
    CL-LOG and delayed logs
    So, in my last post I made untested claims about "specialised binary logs". This week I got to implement one. Interestingly, I only had to make one change to CL-LOG to get this to work: exporting the symbol TIMESTAMP so that code to read binary logs back in can forge backdated timestamps and recreate the original log in text form.

    Slotted this into one of my applications and it just worked. Nice one. Slotted it into another and it tripped over its own feet on startup, about 50% of the time, with a race condition between the new binary messenger and another one which just has to be started up before and just has to run in a different thread. (Don't ask.) Solved this by implementing a "delayed log" messenger - this sits on every message to anywhere until you're ready to let them all go, and the messages then all go out in the right order and with the correct timestamps.

    (defun queue-all-messages ()
  (start-messenger 'queue-messenger
                   :name 'queue-messenger))

(defclass queue-messenger (base-messenger)
  ((queue :reader queue-messenger-queue :initform (make-queue))))

(defmethod print-object ((self queue-messenger) stream)
  (print-unreadable-object (self stream :type t :identity t)
    (format stream "~d" (length (queue-queue (queue-messenger-queue self))))))

(defclass queued-message (base-message)
  ())

(defmethod messenger-send-message :around (messenger (message formatted-message))
  (let* ((log-manager (log-manager))
         (queue-messenger (find 'queue-messenger (log-manager-messengers log-manager) :key 'type-of))
         (queued-message (make-instance 'queued-message
                                        :timestamp :||
                                        :description messenger
                                        :arguments `(,message))))
    (enqueue queued-message (queue-messenger-queue queue-messenger))))

(defmethod messenger-send-message ((self queue-messenger) message)
  nil)

(defmethod stop-messenger :around ((self queue-messenger))
  (let* ((gf #'messenger-send-message)
         (method (find-method gf '(:around) (mapcar 'find-class '(t formatted-message)))))
    (remove-method gf method))
  (call-next-method)
  (dolist (queued-message (queue-queue (queue-messenger-queue self)))
    (messenger-send-message (message-description queued-message)
                            (car (message-arguments queued-message)))))


    (Slot in your own definition of queues, or shove messages onto a list and reverse it.)
    Friday, June 29th, 2007
    6:03 pm
    CL-LOG - a general purpose logging utility

    CL-LOG is a general purpose logging utility, loosely modelled in some respects after Gary King's Log5.

    Its features include:

    • logging to several destinations at once, via "messengers",
    • each messenger is tailored to accept some log messages and reject others, and this tailoring can be changed on-the-fly,
    • very rapid processing of messages which are rejected by all messengers,
    • support for messengers which cl:format text to a stream,
    • support for messengers which do not invoke cl:format,
    • timestamps in theory accurate to internal-time-units-per-second.

    Typically, logging utilities only support the one, simple paradigm of formatting objects to a text-stream. Removing this restriction allows you to:

    • log random objects instead of plain text,
    • generate internationalized log messages,
    • create specialised binary logs,
    • base the test-suite of complex applications around logging.

    Logging is controlled by the following classes: log-manager, base-messenger, base-message; any of these might be specialised by your application. Whether or not a messenger will accept a given message is controlled by the "categories" of both the messenger and message; categories are logical combinations of atoms (typically keywords).

    CL-LOG is released with an MIT-style license.



    Current Music: Another Setting (Durutti Column)
    Saturday, April 14th, 2007
    4:38 pm
    Seeing as I had the first word...
    My perspective on the lisp conference is a bit skewed, given that I chose to go to so few talks. I thoroughly enjoyed Christian Queinnec's talk about teaching Computer Science, Manuel Serrano's about HOP, and Richard Jones' about memory management techniques; I dozed through a couple of others (my fault, not yours!) and I skived off everything else. Just too keyed up and over-exhausted to be able to sit down and concentrate.

    Low point - Monday April 3rd 2006. Edi Weitz and I walked the "conference route", from Clare Old Court through to the Law building. Late morning temperatures about 2 degrees above freezing. Strong feeling that I'd got something wrong.

    Low point - first five days of March, only four registrations.

    High point - March 6th-21st, 53 registrations. On some days (March 9th sticks to mind) it was raining delegates. The final tally was 128.

    High point - day before the conference, trotting through Clare, coming across the college choir rehearsing for a concert that evening.

    High point - cycling from my front door to an international lisp conference.

    Low point - the ALU meeting and subsequent fallout.

    High point - at the next ILC my only involvement is going to be to show up as a delegate.

    High point - the group photo. I'm getting a large copy printed and framed and will keep it on the wall as a memento of some seriously crazy months.

    High point - dressing up for the banquet, and there being someone with a Masters in Classics on High Table to read "Grace".

    High point - knowing how little work there still is to do, and none of it is time critical now.

    High point - that when I go back to work this Monday I'm going to spend the day immersed in lisp rather than in conference stuff.

    High point - meeting all the people I spent months corresponding with.

    Highest point of all - when 100 lispers barreled out of tutorials for morning break on the first day. Seeing everyone in one place really drove home that it had all come together.





    Current Mood: Victorious
    Current Music: End of the Line
    Sunday, April 8th, 2007
    5:35 pm
    Saturday, April 7th, 2007
    2:59 pm
    Lisp sati-fies
    Some evidence of the image on the front of the t-shirts scratching off very readily. Mine is missing part of the second #\s.

    Nobody mentioned this at the conference, so it might just be that the two I took home weren't as good as the others. To be safe, I'd suggest washing or ironing before wearing (if it's not too late...)

    Thursday, April 5th, 2007
    9:44 am
    Spot the lisper


    (click to enlarge)



    Friday, March 30th, 2007
    5:34 pm
    It's Showtime!
    Tracked down the badges, they look great. Late this afternoon the proceedings turned up, they too look great. Previous experience with printed materials warns me not to look at them too carefully. Somebody else can have the honour of spotting the first typo.

    Going to the curry which the LW people have organised. Seeing as it's right next to Castle Hill, I shall do a little climbing first.





    Current Mood: ebullient
    Current Music: Clear White Light
    Thursday, March 29th, 2007
    12:25 pm
    A nice cup of tea and a sit down


    Unbelievably, after nearly two years of planning, it's here. The torrent of emails has dried up, the phone is strangely silent, the t-shirts are piled up in my front room and the printers still claim that the proceedings will be finished and delivered on time. Anything that goes wrong now will have to do so without my help, because the preparations are now officially over. Time for a strong cup of tea and some general unwinding and lazing about, before I go in search of the badges which Christophe has manufactured and the Royal Mail has so thoughtfully borrowed.

    Tomorrow, as it happens, I'm on the road most of the day. I'll be back in town mid afternoon and for me, that's when the conference starts.



    Current Mood: recumbent
    Current Music: Handel's Messiah
About LiveJournal.com