NewLISP Ideas
From newLISP on Noodles
Contents |
newLISP Ideas
This section contains a list of ideas for possible inclusion into future versions of newLISP or the Noodles libraries. Unlike the Code Snippets and Tips and tricks sections, however, it is more intended as a place for discussion (and defense) of ideas. Code is welcome, but it's probably a better idea to include the code in those sections and discuss it here. Older ideas, or ones that have reached some sort of resolution, will be moved to an archive page.
General Guidelines
- Please keep extended discussion to the respective talk pages.
- Please add new ideas at the beginning of the page rather than the end.
Map/Reduce
Map/Reduce[1] seems like an excellent improvement for the distributed side of newLISP. Even better if you could tie it into a DB for smaller data sizes and a cluster filesystem for larger sizes.
--Fischer 09:56, 23 January 2007 (PST)
Scanning over the google paper [2] it seems we have done similar things all year long using net-eval for mapping the same task to different worker nodes in our server cluster and then consolidating reducing to a result and we also where doing it for word counts, similar to the example program in the Google paper. For an addtional abstraction level one could use the newLISP apply function which can take a reduce parameter:
(apply reducer '(data-set-1 data-set-2 data-set-3 ...) 2) => consolidated result-dataset
The function reducer would successively take data sets from different node and then consolidate them. The way newLISP's net-eval is constructed though, using the net-eval idle function may be the better solution in the practice, because data can be processed (reduced) as they arrive:
; the computer cluster nodes (set 'nodes '( (node1 "192.168.02" 4711) (node2 "192.168.0.3 4711) ...) ) ; the documents to process (set 'docs '(doc1 doc2 ...))
; configure a node task ist using 'map' (set 'node-task-list (map (fn (n d) (config task n d)) nodes docs))
; reducer function
(define (idle-func param)
(get-data-from param)
(consolidate-into totals (get-data-from param)))
; distribute tasks using 'net-eval' (net-eval node-task-list timeout idle-func)
I have an example/paper in the works also showing how to use the bayes-train function as an efficient tool to create a word-count namespace (context) with word counts.
Donlucio 07:20, 26 January 2007 (PST)
Here is a complete working example http://newlisp.org/syntax.cgi?code/mapreduce.txt
Donlucio 07:23, 3 March 2007 (PST)
More Matrix Operations
9.0.1 includes several scalar matrix operations. Could these be made more general? Something like (mat <scalar operation> <matrix1> ...) could cover these as well as any scalar operation. Also, is there any hope of removing the 2-dimensional restriction?
Perhaps a matrix.lsp could be included at some point. It could cover a full range of matrix operations, such as determinant, trace, eigenvalues, eigenvectors, decomposition, etc. I'm willing to start this if the basic matrix operations are primitives. I'd rather use newLISP than Mathematica!
--John 16:10, 27 November 2006 (PST)
I don't think much more will be added, a 'det'/determinant is already there since 9.0. LU decompositin is already inside newLISP as part of the 'invert' function. Perhaps it could be brought to the surface as a 'ludecomp' function. That would add very little code.
Beyond that any aspirations to come closer to Mathematica or Matlab in matrix operations represent a slippery slope. newLISP would grow in size too much and loose it's purpose as a smallweight agent language. But I think most of the basics are in place to build a decent math/matrix module around it? Something like stat.lsp.
Donlucio 10:21, 28 November 2006 (PST)
Association Lists
update-assoc is now a candidate in Base.
See the new syntax form of replace in newLISP v.9.0.18 or later, which works like an update-assoc but with much more possibities by using match or unify as optional comparison parameters. For example:
(replace '(x *) data-list replacement match)
would look for all associations in data-list starting with x and execute the function in replacement. The expression (x *) would be compared against each member of data-list using the newLISP match function.
With this method list replacements as complex as regular expression replacements can be performed.
The same approach can be used with find, ref and ref-all. All these functions now can take an optional comparison function like match or unify or any other user-defined function.
Donlucio 17:12, 23 January 2007 (PST)
Included Help
I've been using Python recently (forgive me Lutz) and have come to see the virtues of command line help. I've written a small file based on the syntax for each command. I've included the link and example here only to demonstrate what might become of something like this. The linked file will be out of date as soon as a new version of the manual is generated.
newLISP v.9.0 on OSX, execute 'newlisp -h' for more info. > (silent (load "help.lsp")) > (help "starts-with") syntax: (starts-with list [expr] ) syntax: (starts-with str str-key [num-option] ) "starts-with" > (help 'starts-with) syntax: (starts-with list [expr] ) syntax: (starts-with str str-key [num-option] ) "starts-with" >
--John 15:50, 24 November 2006 (PST)
These things are easy todo in a few lines of newLISP. A (help <func>), which uses the lynx growser to display the whole manual entry in the console, is already in init.lsp.example in the source distribution.
Here is a syntax helper of the form you are describing:
;; syntax-help
;; include this in your init.lsp for syntax commandline help
;; example:
;; > (help starts-with)
;; syntax: (starts-with <str str-key> [<num-option>] )
;; syntax: (starts-with <list> [<expr>] )
;;
;; For Win32 change the manual path accordingly
(define-macro (help)
(let (func-name (name (args 0)))
(dolist (item
(find-all (format {<b>(syntax: \(%s.*?)</b>} func-name)
(read-file "/usr/share/doc/newlisp/newlisp_manual.html") $1)) ; UNIX
(replace "<em>" item "<")
(replace "</em>" item ">")
(println item) func-name)))
;; eof
Quotes are not required for the keyword because define-macro is used:
> (help starts-with) syntax: (starts-with <str str-key> [<num-option>] ) syntax: (starts-with <list> [<expr>] ) "starts-with" >
The macro never needs updating as it extracts information from the currently installed reference manual.
I am not sure where we would put this on the Wiki? Perhaps in the Tips_and_tricks section? We could put all helper fucntions together on a page. Donlucio 06:05, 25 November 2006 (PST)
The above macro help will be included in the source distribution in the examples/ directory in newLISP version 9.1.
Donlucio 14:15, 26 January 2007 (PST)
lookup-all
The latest version of newLISP has ref-all and find-all. Could this be extended to lookup-all and assoc-all? I'd prefer the "all" part to be a parameter to lookup and assoc, but that might break a good bit of existing code.
--John 15:50, 24 November 2006 (PST)
I have a match-all on my list of things to do: match-all would work like this:
(match-all '(? a ?) '((1 2 3) (4 a 5) (a b c) (x a y))) => '((4 a 5) (x a y))
This would be an assoc-all written with match-all:
(match-all '("fo" *) '(("fi" 1 2 3) ("fo" 4 5 6) ("fu" 7 8 9) ("fo" 10 11)) )
=> (("fo" 4 5 6) ("fo" 10 11))
If match-all would take an addtional worker expression, like find-all and replace can already do to process each matched pattern, then a lookup-all could also be modelled easily:
(match-all '(key *) aList (last $0) => (6 11) ; key and aList are defined as previously
The exisiting match routine can discover complex list structures because it can recurse into a list, i.e. ((key ?) *) could also be a valid match pattern. Donlucio 06:30, 25 November 2006 (PST)
See my comments on this same page in the chapter about Asscociation Lists. ref-all with match or unify now works like the above proposed match-all.
Donlucio 17:17, 23 January 2007 (PST)
get version and platform
How about a more user-friendly way to find out what platform you're running?
(if (& 0x7F (sys-info 7))
falls under the category of 'mystic runes'... :-) Cormullion 08:06, 25 November 2006 (PST)
The expression
(& 0x7F (sys-info 7)
just gives you a number between 1 and 9 for the platform running on so for Mac OSX it would be:
(if (= (& 0x7F (sys-info 7) 3)
(do-OSX-stuff))
the 0x7F is used to mask out the bits indicating a UTF-8 version and if a library version. I agree that except for people accustomed to fiddle with bits this is to cryptic. Internally newLISP already carries strings for the different versions to be displayed for the sign-on message. I could stick this string into a ostype variable. Donlucio 11:49, 25 November 2006 (PST)
- good solution - as a confirmed non-bit-twiddler, this would be easier for me! Cormullion 14:08, 25 November 2006 (PST)
