cl-mongo

https://github.com/fons/cl-mongo.git

git clone 'https://github.com/fons/cl-mongo.git'

(ql:quickload :cl-mongo)
98

cl-mongo

Intro

mongo is a scalable, high-performance, open source, schema-free, document-oriented database..

This is a common lisp interface that allows you to work with a mongo document database. This is not a driver (yet) as specified in the mongo documents; There is still some functionality I need to implement.

Except for gridfs most features have been implemented.

cl-mongo provides the ability to insert, update and delete documents, indexing, searching using regexs etc. In addition it supports javascript in the repl (using a commonly available lisp to javascript compiler). This allows you to do use mongodb's map-reduce from the lisp repl.

I developed this primarily using sbcl and to some extend clozure common lisp on the linux and mac osx platforms.

This should also work on clisp and allegro common lisp and windows.

multi-threading

The do-query function uses multithreading to process the results of a batch as a query is in progress. This obviously requires that multi-threading is supported. I've found that as of now (2010-10-23) sbcl on mac osx doesn't support multithreading out of the box. So you'd need to compile a new sbcl with multithreading enabled, which is easy to do.

Version

Version 0.9.0

I've added regression testing for all features currently supported by cl-mongo.

Installation

Use asdf to install cl-mongo. cl-mongo depends on many other lisp libraries (which in turn have their own dependecies). asdf is not a dependency manager, so you would need to install all dependencies as they come up as unavailable when you try to install cl-mongo.

On the other hand, cl-mongo is included in quicklisp, which should make life easier.

Testing

The cl-mongo-test package contains regression tests for all the features currently supported. cl-mongo-test is asdf-installable, but does not export it's tests. To run the quick test, do this : (use-package :cl-mongo-test) (quick-test)

Quick test will connected to a locally running mongodb instance. It uses the “foo” collection in the “test” database.

A sample session

This connects to the test database on the local mongo server listening on its default port.

(use-package :cl-mongo)
(db.use "test")

Insert a key-value pair as the first document into collection “foo”.

(db.insert "foo" (kv "document" "one") )

Pretty-print the documents returned by the find command. iter will ensure that the cursor is fully iterated over.

(pp (iter (db.find "foo" :all)))`

 {
    "_id" -> objectid(4B5CF28970DFF196A75FE1F0)
    "document"  ->  one
 }

Create a document. A document is collection of key-value pairs and a unique identifier called “_id”.

(defvar *DOC* (make-document))

Add various elements to the document.

   (add-element "tag" "key" *DOC*)`


   {  DOCUMENT 

   {
   tag  -> key 
   }
   }


(add-element "array" (list 1 2 3 "hello") *DOC*)

{  DOCUMENT 

{
tag  -> key 
     1 
     2 
     3 
     hello 
 array  -> NIL 
 }
 }

Insert document into the database.

(db.insert "foo" *DOC*)

Print the current contents.

   (pp (iter (db.find "foo" 'all)))

   {
   "_id" -> objectid(4B5CF28970DFF196A75FE1F0)
   "document"  ->  one
   }

   {
   "_id" -> objectid(8B508D5CBB5D451D961F046D)
   "array"  -> [ 1, 2, 3, hello,]
   "tag"  ->  key
   }

Bind variable *DOC* to the second document returned by the find command, add an other element and save back to the collection.

  (defvar *DOC* (cadr (docs (db.find "foo" :all))))`
  (add-element "tags" (list 'good 'bad 'ugly) *DOC*)
  (db.save "foo" *DOC*)
  (pp (db.find "foo" :all))

  {
"_id" -> objectid(4B5CF28970DFF196A75FE1F0)
    "document"  ->  one
  }

  {
    "_id" -> objectid(8B508D5CBB5D451D961F046D)
    "tags"  -> [ GOOD, BAD, UGLY,]
    "tag"  ->  key
    "array"  -> [ 1, 2, 3, hello,]
  }

Check the status of the server.

  (db.use "admin")
  (nd (db.run-command :serverstatus))

    "ok"  ->  1.0d0
  "mem"  -> 
    "mapped"  ->  80
    "virtual"  ->  177
        "resident"  ->  3
    "globalLock"  -> 
      "ratio"  ->  4.644753171959394d-5
      "lockTime"  ->  6493354.0d0
    "totalTime"  ->  1.39799764586d11
    "uptime"  ->  139799.0d0

$ syntax

I've added various $.. macros which allow for a more declarative programming style. In stead of doing something like :

(db.find "foo" (kv "k" (kv "$lte" 3)))

you can use :

(db.find "foo" ($<= "k" 3))

To declare a unique ascending index on “k” in collection “foo” , you can say :

($index "foo" :unique :asc "k")

What's missing

At least the following is missing :

API documentation

I use Edi Weitz's documentation template to generate an api description based on embedded comments.

News

2010-10-23 I've improved the read performance several orders of magnitude. This also reduces the memory foot pront quite a bit.

In addition I've added a multi-threaded reader called do-query.

CL-MONGO - api reference


 

Abstract

The code comes with a MIT-style license so you can basically do with it whatever you want.

Download shortcut: http://github.com/fons/cl-mongo.


 

Contents

  1. Download
  2. The CL-MONGO dictionary
    1. $
    2. $!=
    3. $!in
    4. $+
    5. $-
    6. $/
    7. $<
    8. $⇐
    9. $>
    10. $>=
    11. $add-to-set
    12. $all
    13. $em
    14. $exists
    15. $in
    16. $inc
    17. $index
    18. $map-reduce
    19. $mod
    20. $not
    21. $pop-back
    22. $pop-front
    23. $pull
    24. $pull-all
    25. $push
    26. $push-all
    27. $set
    28. $unset
    29. $where
    30. mongo-default-db
    31. mongo-default-host
    32. mongo-default-port
    33. repo-root
    34. add-element
    35. cwd
    36. date-time
    37. db.add-user
    38. db.auth
    39. db.collections
    40. db.collections
    41. db.count
    42. db.create-collection
    43. db.delete
    44. db.distinct
    45. db.ensure-index
    46. db.eval
    47. db.find
    48. db.indexes
    49. db.indexes
    50. db.insert
    51. db.iter
    52. db.next
    53. db.run-command
    54. db.save
    55. db.sort
    56. db.stop
    57. db.update
    58. db.use
    59. defjs
    60. defsrvjs
    61. do-query
    62. doc-id
    63. docs
    64. document
    65. generate-readme
    66. get-element
    67. ht→document
    68. install-js
    69. iter
    70. jsdef
    71. kv
    72. make-document
    73. mapdoc
    74. mongo
    75. mongo
    76. mongo-close
    77. mongo-registered
    78. mongo-show
    79. mongo-swap
    80. mr.gc
    81. mr.gc.all
    82. mr.p
    83. nd
    84. now
    85. nwd
    86. pp
    87. remove-js
    88. ret
    89. rm
    90. rm-element
    91. show
    92. time-zone
    93. with-mongo-connection
  3. Acknowledgements


 

Download

CL-MONGO together with this documentation can be downloaded from http://github.com/fons/cl-mongo. The current version is 0.1.0.


 

The CL-MONGO dictionary


[Macro]
$ &rest args => result



[Macro]
$!= &rest args => result



[Macro]
$!in &rest args => result



[Macro]
$+ arg &rest args => result



[Macro]
$- arg &rest args => result



[Macro]
$/ regex options => result



[Macro]
$< &rest args => result



[Macro]
$⇐ &rest args => result



[Macro]
$> &rest args => result



[Macro]
$>= &rest args => result



[Macro]
$add-to-set &rest args => result



[Macro]
$all &rest args => result



[Macro]
$em array &rest args => result



[Macro]
$exists &rest args => result



[Macro]
$in &rest args => result



[Macro]
$inc &rest args => result



[Macro]
$index collection &rest args => result



[Macro]
$map-reduce collection map reduce &key query limit out keeptemp finalize verbose => result


Run map reduce on the mongo server. map and reduce are either the names of the javascript functions, created with defjs or defsrvjs or are function definitions in javascript. The keywords refer to option available for map reduce in mongo. This returns a result summary document. When using :keeptemp t without specificing :out the collection is mr.<collection>


[Macro]
$mod &rest args => result



[Macro]
$not &rest args => result



[Macro]
$pop-back &rest args => result



[Macro]
$pop-front &rest args => result



[Macro]
$pull &rest args => result



[Macro]
$pull-all &rest args => result



[Macro]
$push &rest args => result



[Macro]
$push-all &rest args => result



[Macro]
$set &rest args => result



[Macro]
$unset &rest args => result



[Macro]
$where &rest args => result



[Special variable]
mongo-default-db


database opened by the default connection


[Special variable]
mongo-default-host


host for the default connection.


[Special variable]
mongo-default-port


port for the default connection.


[Special variable]
repo-root


root of the repository; used for documentation generation


[Generic function]
add-element key value document => result


add element with key and value to a document


[Function]
cwd &key mongo => result


Show the current database.


[Function]
date-time second minute hour day month year &optional time-zone => result


Generate a time stamp the mongo/bson protocol understands.


[Generic function]
db.add-user username password &key mongo readonly => result


Add a user to the database.


[Generic function]
db.auth username password &key mongo => result


authenticate a user with a password


[Generic function]
db.collections &key mongo => result



[Function]
db.collections &key mongo => result


Show all the collections in the current database.


[Generic function]
db.count collection selector &key mongo => result


Count all the collections satifying the criterion set by the selector. :all can be used to return a count of all the documents in the collection.


[Generic function]
db.create-collection collection &key => result


create a collection


[Generic function]
db.delete collection object &key mongo => result


Delete a document from a collection. The document field is used to identify the document to be deleted.
You can enter a list of documents. In that the server will be contacted to delete each one of these. It may be more efficient to run a delete script on the server side.


[Generic function]
db.distinct collection key &key mongo => result


Return all the distinct values of this key in the collection


[Generic function]
db.ensure-index collection keys &key drop-duplicates unique mongo asc => result


Create an index specified by the keys in a collection


[Generic function]
db.eval code &rest rest => result


run javascript code server side


[Generic function]
db.find collection kv &key selector limit skip options mongo => result


Find documents in the collection using the selector specified by kv.
Methods take two keywords. ':limit' sets the maximum number of documents returned. The default is 1. ':skip' sets the number of documents to skip in this query. It's default is 0. Since the default value of the limit is one, db.find by default is the equivalant of findOne in the mongo documentation.


[Generic function]
db.indexes &key mongo => result



[Function]
db.indexes &key mongo => result


Return all indexes in the database.


[Generic function]
db.insert collection document &key mongo => result


Insert a document in a collection. A document is typically generated by (make-document), but it can also be a hash table, a key-value pair or kv list (see the kv functions).


[Generic function]
db.iter result &key limit mongo => result


next document iteration


[Generic function]
db.next collection cursor-id &key limit mongo => result


Executes the next call on the iterator identified by cursor-id.


[Generic function]
db.run-command cmd &key arg mongo collection index => result


Run a database command on the server. See the mongo documentation for a list of commands.
For most commands you can just uses the key-value shown in the mongo documentation.


[Generic function]
db.save collection document &key mongo => result


Save a document to the collection. If the document has a unique _id value (i.e. if it's generated by (make-document) ) it will be 'upserted' (that is: it will be inserted if the document doesn't exist). If the document a hash table or a kv set, it will be inserted.
In other words this a a helper-function build around db.insert and db.update.


[Macro]
db.sort collection query &rest args => result


sort macro : Takes the same arguments and keywords as db.find but converts the query so it works as a sort. use the :field keyword to select the field to sort on. Set :asc to nil to reverse the sort order


[Generic function]
db.stop cursor &key mongo => result


Stop iterating and clean up the iterator on the server by making a server call.


[Generic function]
db.update collection selector new-document &key mongo upsert multi => result


In a collection update the document(s) identified by the selector statement.
This method has two keywords. ':upsert' : If t insert the document if the document cannot be found in the collection. ':multi' : Update all documents identified by the selector.


[Generic function]
db.use db &key mongo => result


Use a database on the mongo server. Opens a connection if one isn't already established. (db.use -) can be used to go to a previosuly visited database, similar to cd -.


[Macro]
defjs name args declaration* statement* => result


Define client side javascript. Works like defun; body is in lisp, with parenscript additions, like return. So (defjs hello (x y) (return (+ x y))) defines an adder. macro creates a lisp function which sends the javascript function over to the mongo server to be evaluated. Result is processed and returned to the reader. This will execute 10 times on the server : (mapcar (lambda (x) (hello 10 x)) (list 1 2 3 4 5 6 7 8 9 10))


[Macro]
defsrvjs name args declaration* statement* => result


Creates a function which stores and executes javascript on the server. The first time the function is called the javascript function is stored on the server. Subsequent calls will call out to the server. Works like defun; the function body is defined in lisp, with parenscript additions. Since the body of the function already resides on the server this should have less impact on the network. Use :install t to reinstall.


[Function]
do-query coll &key map-fn reduce-fn initial-value query mongo limit selector => result


Performs a multi-threaded query on a mongo database. coll is the collection name. The reduce-fn keyword is used to specify a function which will be called for each member of a batch of data returned from mongodb.
The reduce-fn function is executed while the query for the next batch is in progress. The default for reduce-fn is the identity function. The reduction keyword is used to specify a function which is executed when the database queries have finished. It's default implementation is to return a hash table, with the mongodb id as the hash key. The query, mongo, limit and selector keywords are used in the same way as for db.find.


[Function]
doc-id doc => result


return the unique document id


[Function]
docs result => result


Stop the iterator (if any) and return the list of documents returned by the query. Typical ue would be in conjunction with db.find like so (docs (iter (db.find 'foo' 'll)))


[Standard class]
document


document Document class. A document consists of key/value pairs stored in a internal hash table plus an internally generated unique id.
Accessors are : 'elements' which returns the internal hash table; 'id' which returns the unique id and 'local_id' which if true means that the document was generated by the client (as opposed to having been read from the server).


[Function]
generate-readme &key path => result


This function generates a README.md file with the latest api description. The :path keyword specifies the location. It expects a sub-directory <path>/doc. Api documentation is generated on the fly, by extracting comments from the classes, generics and fuctions exported in the packages file. The resulting file is <path>/doc/index.html. <path>/README.md is generated by
appending the api documentation to <path>/doc/readme-base.md. :path or REPO-ROOT are typically set to the root of the repository.


[Generic function]
get-element key document => result


Get an element identified by key from the document.


[Function]
ht→document ht => result


Convert a hash-table to a document.


[Macro]
install-js name => result


Allows server based javascripts the be installed without being run.


[Function]
iter result &key mongo max-per-call => result


Exhaustively iterate through a query. The maximum number of responses per query can be specified using the max-per-call keyword.


[Macro]
jsdef name => result


Return the body of the javascript function; otherwise nill.


[Generic function]
kv a &rest rest => result


This a helper function for key-value pairs and sets of key-value pairs.
In a key-value pair like (kv key value) the key has to be a string and the value something which is serializable. key-value pairs can be combined using kv as well : (kv (kv key1 val1) (kv key2 val2)). This combination of key-value pairs is equivalent to a document without a unique id. The server will assign a unique is if a list of key-value pairs is saved.


[Function]
make-document &key oid size => result


Constructor. key ':oid' is a user supplied unique id. An internal id will be generated if none is supplied.


[Function]
mapdoc fn document => result



[Standard class]
mongo


Encapsulates the connection to the mongo database. Each connection is a added to a global registry.


[Generic function]
mongo &key host port db name host port db name => result


This method returns the connection referred to by the name identifier from the connection registry. The connection name is unique. If no connection with that name exists, a new connection with the supplied or default host, port and db parameters will be created. The default host is localhost; the default port is 27017; the default db is admin.


[Generic function]
mongo-close name => result


Close the connection to the mongo database. The name should uniquely identify the connection to close. This is either a mongo object or the name the object is bound to in the connection registry. To close all open connections use the special symbol 'all


[Generic function]
mongo-registered name => result


Return a conection registered by this name or nil..


[Function]
mongo-show => result


Show all registered connections and their session id


[Generic function]
mongo-swap left right => result


Swap the names of the left and right connections. Typical use would be (mongo-swap :default :alt). After the function call :default will refer to the connection previously referred to as :alt. A connection named :default is returned by (mongo) and is the default used in the api. The connections are returned in the order they were passed in (but with the names swapped between them). To re-open a connection you can say (mongo-close (mongo-swap :default (mongo :host &lt;newhost&gt; :portid &lt;portid&gt; :name :temp))) and a new default connection is registered.


[Function]
mr.gc &key mongo => result


remove the temporary collections created by map-reduce


[Function]
mr.gc.all &key mongo => result


remove the all collections created by map-reduce, temporary as well as permanent


[Function]
mr.p results => result


show the contents of the results collection map-reduce created


[Function]
nd result &key stream => result


Pretty-print for non-document responses, like the response to a database command.


[Function]
now => result


Return the current date and time in bson format.


[Function]
nwd => result


Show the database set by the (db.use -) command


[Generic function]
pp result &key nd stream => result


Pretty-print the results returned from a query. To be used on the repl. This will format each server reply as if it were a document. This is obviously ok in mosty cases. See nd for an alternative.


[Macro]
remove-js name => result



[Function]
ret result => result


return value bound to retval in a return document. Used for db.count, db.distinct, functions etc..


[Function]
rm collection query &key mongo => result


Delete all the documents returned by a query. This is not an efficient way of deleting documents as it invloves multiple trips to the server. Mongo allows for execution of java-script on the server side, which provides an alternative. Typical use would be (rm (iter (db.find 'foo' (kv 'key' 1)))), which deletes all documents in foo, with field key equal to 1.


[Generic function]
rm-element key document => result


Remove element identified by key from a document


[Generic function]
show things &key msg nl order => result


Print a list of things. Things can be users, databases, collections in the current database, the profile and more. Things is a keyword so (show :users) will show all users.


[Function]
time-zone => result


Set the time zone appropriate for the current environment.


[Macro]
with-mongo-connection args &rest body => result


Creates a connection to a mongodb, makes it the default connection and evaluates the body form. args uses the same keyword set as mongo (:db :localhost :port) args is passed on to make-mongo when the connection is created.


 

Acknowledgements

This documentation was prepared with DOCUMENTATION-TEMPLATE.

$Header: /usr/local/cvsrep/documentation-template/output.lisp,v 1.14 2008/05/29 08:23:37 edi Exp $

BACK TO MY HOMEPAGE