The Jim Interpreter » News

Friday, 10 February 2012

Coping without Namespaces - Revisited

Back in the article Coping without Namespaces, we discussed that fact that Jim Tcl did not support namespaces, but that small changes to the source could be made to allow namespace-based Tcl code to be ported for use with Jim Tcl.

Now with namespaces supported in Jim Tcl 0.73, porting Tcl code which makes use of namespaces is easier than ever.

Once again, let's consider porting dns.tcl from tcllib to Jim Tcl.

Firstly an explanation of what was changed.

--- dns.tcl.orig	2012-03-05 13:02:36.000000000 +1000
+++ dns.tcl	2012-03-05 13:02:56.000000000 +1000
@@ -1,3 +1,15 @@
+# dns.tcl - Steve Bennett <steveb@workware.net.au>
+#
+# Modified for Jim Tcl to:
+# - use udp transport by default
+# - use sendto/recvfrom
+# - don't try to determine local nameservers
+# - remove support for dns uris and finding local nameservers
+# - remove logging calls
+#   (both of these in order to remove dependencies on tcllib)
+
+# Based on:
+
 # dns.tcl - Copyright (C) 2002 Pat Thoyts <patthoyts@users.sourceforge.net>
 #
 # Provide a Tcl only Domain Name Service client. See RFC 1034 and RFC 1035

For simplicity, we remove the dependencies on tcllib logger, uri and ip to allow this example to be self contained. In Jim Tcl the binary and namespace modules are optional, so load them if required.

@@ -31,14 +43,11 @@
 #
 # $Id: dns.tcl,v 1.36 2008/11/22 12:28:54 mic42 Exp $
 
-package require Tcl 8.2;                # tcl minimum version
-package require logger;                 # tcllib 1.3
-package require uri;                    # tcllib 1.1
-package require uri::urn;               # tcllib 1.2
-package require ip;                     # tcllib 1.7
+package require binary
+package require namespace
 
 namespace eval ::dns {
-    variable version 1.3.3
+    variable version 1.3.3-jim2
     variable rcsid {$Id: dns.tcl,v 1.36 2008/11/22 12:28:54 mic42 Exp $}
 
     namespace export configure resolve name address cname \

Since Jim Tcl supports udp out-of-the-box, and it is more efficient, default to udp rather than tcp. Also comment out the logging calls.

@@ -49,23 +58,13 @@
         array set options {
             port       53
             timeout    30000
-            protocol   tcp
+            protocol   udp
             search     {}
             nameserver {localhost}
             loglevel   warn
         }
-        variable log [logger::init dns]
-        ${log}::setlevel $options(loglevel)
-    }
-
-    # We can use either ceptcl or tcludp for UDP support.
-    if {![catch {package require udp 1.0.4} msg]} { ;# tcludp 1.0.4+
-        # If TclUDP 1.0.4 or better is available, use it.
-        set options(protocol) udp
-    } else {
-        if {![catch {package require ceptcl} msg]} {
-            set options(protocol) udp
-        }
+        #variable log [logger::init dns]
+        #${log}::setlevel $options(loglevel)
     }
 
     variable types

udp is built-in with Jim Tcl.

@@ -248,14 +237,6 @@
         return -code error "no nameserver specified"
     }
 
-    if {$state(-protocol) == "udp"} {
-        if {[llength [package provide ceptcl]] == 0 \
-                && [llength [package provide udp]] == 0} {
-            return -code error "udp support is not available,\
-                get ceptcl or tcludp"
-        }
-    }
-    
     # Check for reverse lookups
     if {[regexp {^(?:\d{0,3}\.){3}\d{0,3}$} $state(query)]} {
         set addr [lreverse [split $state(query) .]]

udp in Jim Tcl works just like tcp, with readable event handler being triggered when the response is available.

@@ -273,6 +254,7 @@
         }
     } else {
         UdpTransmit $token
+        wait $token
     }
     
     return $token

Jim Tcl has no support for async connect, and the parameters to socket are a little different.

@@ -668,9 +650,9 @@
                                    "operation timed out"]]
     }
 
-    # Sometimes DNS servers drop TCP requests. So it's better to
-    # use asynchronous connect
-    set s [socket -async $state(-nameserver) $state(-port)]
+    # Jim Tcl has no async connect ...
+
+    set s [socket stream $state(-nameserver):$state(-port)]
     fileevent $s writable [list [namespace origin TcpConnected] $token $s]
     set state(sock) $s
     set state(status) connect

Comment out the async connect check.

@@ -683,11 +665,13 @@
     upvar 0 $token state
 
     fileevent $s writable {}
-    if {[catch {fconfigure $s -peername}]} {
-	# TCP connection failed
-        Finish $token "can't connect to server"
-	return
-    }
+
+    # Jim Tcl has no async connect ...
+#    if {[catch {fconfigure $s -peername}]} {
+#	# TCP connection failed
+#        Finish $token "can't connect to server"
+#	return
+#    }
 
     fconfigure $s -blocking 0 -translation binary -buffering none

udp in Jim Tcl is easy. Simpy create the socket with socket dgram and send with sendto.

@@ -722,18 +706,10 @@
                                   "operation timed out"]]
     }
     
-    if {[llength [package provide ceptcl]] > 0} {
-        # using ceptcl
-        set state(sock) [cep -type datagram $state(-nameserver) $state(-port)]
-        fconfigure $state(sock) -blocking 0
-    } else {
-        # using tcludp
-        set state(sock) [udp_open]
-        udp_conf $state(sock) $state(-nameserver) $state(-port)
-    }
-    fconfigure $state(sock) -translation binary -buffering none
+    set state(sock) [socket dgram]
+    #fconfigure $state(sock) -translation binary -buffering none
     set state(status) connect
-    puts -nonewline $state(sock) $state(request)
+    $state(sock) sendto $state(request) $state(-nameserver):$state(-port)
     
     fileevent $state(sock) readable [list [namespace current]::UdpEvent $token]

Reading from a udp socket is best done with recvfrom

@@ -879,7 +855,7 @@
     upvar 0 $token state
     set s $state(sock)
 
-    set payload [read $state(sock)]
+    set payload [$state(sock) recvfrom 1500]
     append state(reply) $payload
 
     binary scan $payload S id

Jim Tcl has lreverse built-in

@@ -1011,17 +987,6 @@
 }
 
 # -------------------------------------------------------------------------
-# Description:
-#   Reverse a list. Code from http://wiki.tcl.tk/tcl/43
-#
-proc ::dns::lreverse {lst} {
-    set res {}
-    set i [llength $lst]
-    while {$i} {lappend res [lindex $lst [incr i -1]]}
-    return $res
-}
-
-# -------------------------------------------------------------------------
 
 proc ::dns::KeyOf {arrayname value {default {}}} {
     upvar $arrayname array

Notice that no namespace-related changes were required when porting this module.

The latest version of dns.tcl for Jim Tcl is available in git.

Steve Bennett (steveb@workware.net.au)

Comments >>>

Thursday, 22 December 2011

Jim Tcl version 0.73

ANNOUNCE: Jim Tcl version 0.73

Jim Tcl 0.73 has been released and is available from:

http://repo.or.cz/w/jimtcl.git or https://github.com/msteveb/jimtcl

Find out all about Jim Tcl at http://jim.tcl.tk/

CHANGES SINCE VERSION 0.72

This release incorporates bug fixes and many new features. A summary is below. See git for the full changelog.

Bugs fixed in version 0.73

  • exec on cygwin now correctly passes $::env
  • On mingw and cygwin, --shared creates libjim.dll rather than libjim.so
  • UTF-8 case folding may change the encoded length
  • Fix a regexp infinite loop on invalid UTF-8 strings
  • Prevent infinite recursion in eval
  • Don’t allow upvar to a higher level
  • regexp counted matches may be wrong on subsequent use
  • Form feed (\f) is a valid white space character
  • Parsing bug for quoted orphan $
  • Standard handles were not being kept open
  • Fix edge cases with tailcall

Features added in version 0.73

  • Added support for namespaces and the namespace command
  • The “full” sqlite3 extension is included in the repo
  • Built-in regexp now support non-capturing parentheses: (?:...)
  • Added string replace and string totitle
  • Added info statics to access proc static variables
  • Added info alias to access the target of an alias
  • Added build-jim-ext for easy separate building of loadable modules (extensions)
  • local now works with any command, not just procs
  • UTF-8 encoding past the basic multilingual plane (BMP) is supported
  • Added tcl::prefix
  • Added the history command to access command line editing and history from scripts
  • Added a Tcl-compatible apply command
  • Most extensions are now enabled by default
  • Jim Tcl now compiles with MSVC on Windows (but no build support)

Steve Bennett (steveb@workware.net.au)

Comments >>>

Thursday, 15 December 2011

Lightweight Namespaces

New in Jim Tcl v0.73 is (optional) support for namespaces.

The following is taken directly from README.namespaces in the Jim Tcl repository.

Lightweight Namespaces for Jim Tcl

There are two broad requirements for namespace support in Jim Tcl.

  • (1) To allow code from multiple sources while reducing the chance of name clashes
  • (2) To simplify porting existing Tcl code which uses namespaces

This proposal addresses both of these requirements, with the following additional requirements imposed by Jim Tcl.

  • (3) Support for namespaces should be optional, with the space and time overhead when namespaces are disabled as close to zero as possible.
  • (4) The implementation should be small and reasonably efficient.

To further expand on requirement (2), the goal is not to be able to run any Tcl scripts using namespaces with no changes. Rather, scripts which use namespaces in a straightforward manner, should be easily ported with changes which are compatible with Tcl.

Implicit namespaces

Rather than supporting explicit namespaces as Tcl does, Jim Tcl supports implicit namespaces. Any procedure or variable which is defined with a name containing ::, is implicitly scoped within a namespace.

For example, the following procedure and variable are created in the namespace ‘test’

proc ::test::myproc {} {
  puts "I am in namespace [namespace current]"
}
set ::test::myvar 3

This approach allows much of the existing variable and command resolution machinery to be used with little change. It also means that it is possible to simply define a namespace-scoped variable or procedure without first creating the namespace, and similarly, namespaces “disappear” when all variables and procedures defined with the namespace scope are deleted.

Namespaces, procedures and call frames

When namespace support is enabled (at build time), each procedure has an associated namespace (based on the procedure name). When the procedure is evaluated, the namespace for the created call frame is set to the namespace associated with the procedure.

Command resolution is based on the namespace of the current call frame. An unscoped command name will first be looked up in the current namespace, and then in the global namespace.

This also means that commands which do not create a call frame (such as commands implemented in C) do not have an associated namespace.

Similarly to Tcl, namespace eval introduces a temporary, anonymous call frame with the associated namespace. For example, the following will return “::test,1”.

namespace eval test {
	puts [namespace current],[info level]
}

Variable resolution

The variable command in Jim Tcl has the same syntax as Tcl, but is closer in behaviour to the global command. The variable command creates a link from a local variable to a namespace variable, possibly initialising it.

For example, the following procedure uses ‘variable’ to initialse and access myvar.

proc ::test::myproc {} {
  variable myvar 4
  incr myvar
}

Note that there is no automatic resolution of namespace variables. For example, the following will not work.

namespace eval ::test {
  variable myvar 4
}
namespace eval ::test {
  # This will increment a local variable, not ::test::myvar
  incr myvar
}

And similarly, the following will only access local variables

set x 3
namespace eval ::test {
	# This will incremement a local variable, not ::x
	incr x
	# This will also increment a local variable
	incr abc::def
}

In the same way that variable resolution does not “fall back” to global variables, it also does not “fall back” to namespace variables.

This approach allows name resolution to be simpler and more efficient since it uses the same variable linking mechanism as upvar/global and it allows namespaces to be implicit. It also solves the “creative writing” problem where a variable may be created in an unintentional scope.

The namespace command

Currently, the following namespace commands are supported.

  • current - returns the current, fully-qualified namespace
  • eval - evaluates a script in a namespace (introduces a call frame)
  • qualifiers, tail, parent - note that these do not check for existence
  • code, inscope - implemented
  • delete - deletes all variables and commands with the namespace prefix
  • which - implemented
  • upvar - implemented

namespace children, exists, path

With implicit namespaces, the namespace exists and namespace children commands are expensive to implement and are of limited use. Checking the existence of a namespace can be better done by checking for the existence of a known procedure or variable in the namespace.

Command resolution is always done by first looking in the namespace and then at the global scope, so namespace path is not required.

namespace ensemble

The namespace ensemble command is not currently supported. A future version of Jim Tcl will have a general-purpose ensemble creation and manipulation mechanism and namespace ensemble will be implemented in terms of that mechanism.

namespace import, export, forget, origin

Since Jim Tcl namespaces are implicit, there is no location to store export patterns. Therefore the namespace export command is a dummy command which does nothing. All procedures in a namespace are considered to be exported.

The namespace import command works by creating aliases to the target namespace procedures.

namespace forget is not implemented.

namespace origin understands aliases created by namespace import and can return the original command.

namespace unknown

If an undefined command is invoked, the “unknown” command is invoked. The same namespace resolution rules apply for the unknown command. This means that in the following example, test::unknown will be invoked for the missing command rather than the global ::unknown.

proc unknown {args} {
	puts "global unknown"
}

proc test::unknown {args} {
	puts "test unknown"
}

namespace eval test {
	bogus
}

This approach requires no special support and provides enough flexibility that the namespace unknown command is not implemented.

Porting namespace code from Tcl to Jim Tcl

For most code, the following changes will be sufficient to port code.

  1. Canonicalise namespace names. For example, ::ns:: should be written as ::ns or ns as appropriate, and excess colons should be removed. For example ::ns:::blah should be written as ::ns::blah (Note that the only “excess colon” case supported is ::::abc in order to support [namespace current]::abc in the global namespace)

  2. The variable command should be used within namespace eval to link to namespace variables, and access to variables in other namespaces should be fully qualified

Changes in the core Jim Tcl

Previously Jim Tcl performed no scoping of command names. i.e. The ::format command was considered different from the format command.

Even if namespace support is disabled, the command resolution will recognised global scoping of commands and treat these as identical.

Comments >>>

Thursday, 10 November 2011

Jim @ Tcl/Tk 2011

At the Tck/Tk 2011 conference I presented a paper on Jim Tcl. It seemed to be well received.

You can read the paper by clicking on the image below.

Steve Bennett (steveb@workware.net.au)

Comments >>>

Thursday, 06 October 2011

Jim Tcl version 0.72

ANNOUNCE: Jim Tcl version 0.72

Jim Tcl 0.72 has been released and is available from:

http://repo.or.cz/w/jimtcl.git or https://github.com/msteveb/jimtcl

CHANGES SINCE VERSION 0.71

This release incorporates bug fixes and many new features.

Bugs fixed in version 0.72

  • Improvements to configure (autosetup)
  • Fix memory overwrite in built-in regexp
  • [regexp], [regsub] could leak objects in some circumstances
  • Fix [file join] in some cases
  • [info nameofexecutable] now always returns an absolute path
  • [catch] works correctly for platforms without long long
  • [dict unset] no longer gives an error on missing last key

Features added in version 0.72

  • [proc] now accepts optional parameters and “args” in any position
  • Much improved support for mingw32 including 64-bit mingw
    • tcl_platform(platform) is set to “windows”
    • [file dirname] handles leading drive (e.g. c:/)
    • [pwd] returns a path containing only forward slashes
    • [glob] now works correctly
    • [exec] is now fully implement on mingw32
    • Command line editing now works in the win32 console
    • New tcl_platform(pathSeparator)
  • Builtin regexp is more efficient. Patterns are only compiled once.
  • Add rand(), srand() and pow() math functions
  • [file delete] now supports the -force option
  • [fconfigure -translation] is now accepted and ignored for Tcl compatibility
  • Improved diagnostics when sourcing a script with missing/mismatches brackets, quotes, etc.
  • Jim Tcl now builds on Haiku (BeOS clone) and Solaris
  • Build now works with BSD make
  • [file mtime] can now set the file time
  • New [aio listen] to set the size of the listen queue on server sockets
  • Jim Tcl Manual is better formatted, commands are hyperlinked and various corrections have been made
  • The oo, tree, binary and pack extensions are now documented
  • New metakit extension
  • The SDL extension once again builds and runs

Steve Bennett (steveb@workware.net.au)

Comments >>>

Tuesday, 04 October 2011

jSQLsh - Jim Tcl SQLite Shell

The jSQLsh project at https://github.com/LStinson/jSQLsh provides a small but powerful shell for sqlite, modelled after psql

The creator, Lorance Stinson, wanted an sqlite command shell which addressed some of the deficiencies of the native sqlite command shell, so he decided to create his own with Jim Tcl.

To run it, you simply need jimsh built with sqlite3 and readline support. For example, to build a version of jimsh with these extensions included statically:

$ ./configure --with-ext="sqlite3 rlprompt"
...
$ make
...

And then try it out:

$ ./jimsh jsqlsh
Welcome to the SQLite Shell in Jim TCL 0.72.
Opening the database ':memory:'.
To execute a query type '/'. For help type '/h'.
:memory: (0 rows, 0 changes) # /h
Commands:
  /A(UTO)     Toggle the auto option state.
  /c(lear)    Clear the query buffer
  /D(EBUG)    Toggle the debug option state.
  /d          Display all objects in the database.
  /d[itv]     Display all indexes (/di) / tables (/dt) / views (/dv).
  /d OBJ      Describe the object OBJ.
  /ds OBJ     Displays the schema for the object OBJ.
  /e(dit)     Edit the query buffer.
  /go | /     Execute the query in the query buffer.
  /h(elp)     Print this help text.
  /o(pen)     Open a database file, change directories and list files.
  /P(AGER)    Toggle the page option state.
  /p(rint)    Print the query buffer.
  /s(et)      Set/List configuration options. '/s OPTION VALUE'
              Values are treated as TCL strings and can be quoted.
  /Q(uiet)    Toggle the quiet option state.
  /q(uit)     Quit  (Also Ctrl-D)
  /u(ser)     Display the user macros.
  /u(ser)#    Copy user macro # into the query buffer.
  /u(ser)# -  Copy the query buffer, or supplied text, into macro #.
:memory: (0 rows, 0 changes) #

Steve Bennett (steveb@workware.net.au)

Comments >>>

See All News Articles »