LiveCode Forums

I've been looking into integrating the mongodb library.
So far there's no problem compiling the mongo C library and using it as a LiveCode external.
But serious problems with the inflexibility of the revdb database structure.

So I'd like some guidance on what the future of the revdb handlers might be. If this is being reworked now, or if there are plans to rework the way this works in the near future, then I'll shelve this for now. (and I'm hoping that the fabled db rework is in the plans).

But if that's nothing that's being worked on then I'll need to modify the higher-level revdb files to accomodate mongo, and similar work would need to be done to extend this to other database types.

I think the DB rework is to be completed after the new externals API

@mwieder: The revDB architecture is a little antiquated - however one thing to bear in mind is that it was designed to be a wrapper around SQL databases rather than other database types and the architecture and API reflect this.

I must confess I know little about 'NoSQL' databases, but from a cursory glance at the mongo C API it does seem quite different from SQL databases so I guess one question to ask is whether it makes sense to try and shoehorn this kind of database into the revDB driver system, after all if you end up having to add an almost completely distinct set of APIs for it then it would probably better sit as an entirely separate entity.

Do you know whether other languages attempt to abstract database APIs that work on both SQL and NoSQL databases? Are there other NoSQL databases with a similar API to access as MongoDB?

I'm with RunRevMark: let MongoDB be MongoDB, and make a unique interface for that unique data store.

If consistency with any other system is useful, it might be with CouchDB - are you planning on writing an interface for that too?

If so, way cool, but if not just do whatever you feel is most useful in the MongoDB way.

@mwieder: I realized after coming back to this that I didn't really answer your question...

In terms of the revDB rework, the plan is to replace revDB with a new (low-level) abstraction layer wrapped in proper syntax (we'll have Open Language by then). The design will basically have an underlying C library that abstracts away from driver specific details (just as revDB does), but integrates properly with arrays, unicode and binary strings (all of which would be troublesome to do with revDB due to the need to support existing code using it).

I think the question comes down to whether MongoDB and similar NoSQL-type database access can be unified with SQL-type database access at the same level. (At this point, I don't know the answer to this question as I've - personally - not looked at a comparison between them in enough depth).

Ultimately I'd like to see a much higher level of abstraction around databases that moves away from the requirement of using SQL queries for CRUD type operations - however, such a level would likely be built upon a revDB type abstraction. It could be that unifying NoSQL and SQL database access would make more sense at that point...

The points about mongodb being a nosql database are fine, although there are ways to map sql queries to executable nosql statements. Or at least *some* sql queries. And these are detailed a bit on the mongo web site - sort of an intro to nosql databases for sql programmers.

If I have to choose between creating a mongodb external that has no commonality with the existing database engine and trying to shoehorn what can be adapted to the existing engine, I'll put this off for a while. I don't see a point in making something that doesn't touch base with any other database commands, and I won't even be able to leverage things like revOpenDatabase. I think that would just lead to a balkanization of commands for other databases (revOpenMongoDB, revOpenCouchDB, etc). What I think I'd like to see is the ability to execute open, close, execute, and return query results in a generic manner, abstracted from the current need to query the external db library for vector points. I can't fit the mongodb library into that format right now without either rewriting the mongodb makefiles or the revdb engine or both.

How many people interesting in using MongoDB need to be able to switch to a different database with compatible syntax?

Let SQL be SQL, let NoSQL be NoSQL.

Well, for one thing, switching to a nosql database might be less of a challenge if we did have a more compatible syntax.
For another, a more generic nosql syntax would make more sense to me than separate syntaxes for each type of nosql database.
Are you proposing that we have revOpenMySQLDatabase, revOpenODBCDatabase, etc?

How many people interesting in using MongoDB need to be able to switch to a different database with compatible syntax?

I'm not sure that's what @mwieder is saying... It isn't so much about writing script against MongoDB and then just being able to use the same script against MySQL - that clearly wouldn't be possible (unless one was to write an SQL query emulator for NoSQL databases) - it's about unifying the abstractions that can be unified, and specializing for the rest. Some operations that would appear to be common would be:

• Opening / connecting to the database
• Managing the connection 'id'
• Closing the database
• Creating / managing / destroying a query object
• Fetching the results of a query object, and indeed the format of the results

It seems sensible to me that if the syntax for these operations is the same, then it means once you've learnt the shared syntax you can read code that uses any type of database in general, even if you've never used a NoSQL database you'd still be able to get the jist of what was going on.

Indeed, you could imagine an interface hierarchy - there is a the notion of the Database at the root which is all the generic/house-keeping type syntax, then from this you have specializations NoSQLDatabase and SQLDatabase providing common functionality for those database types, then specific specializations for each database type MySQLDatabase, MongoDBDatabase.

For example, assuming there is a notion of 'query string' for NoSQL databases (as there is for SQL databases), then there is a generic Query operation - you give it a string which the concrete database understands and it returns you a cursor which can then be manipulated to fetch the results.

[ I think one important thing to point out is that you can't just switch the database type field in revOpenDatabase at the moment and have things just work, except for perhaps the simplest of applications. There are enough variance in all the major SQL databases that you do need to design your app to work with, say, PostgreSQL and MySQL if that is what you want. At the 'revDB' level things are still too low-level to abstract away database-specific differences (although I do admit revDB could do things a little better - in regards to typing and database structure queries) - for that level of database-portability I do think a higher-level abstraction required. ]

Are you proposing that we have revOpenMySQLDatabase, revOpenODBCDatabase, etc?

To be fair this is essentially the case already: But I don't think the opening case is really the interesting one, as specific database types are always going to have specific connection requirements. I think the interesting part of unification is that which it gives to processing results, and managing the cursors and result sets in a uniform way.

It seems sensible to me that if the syntax for these operations is the same, then it means once you've learnt the shared syntax you can read code that uses any type of database in general, even if you've never used a NoSQL database you'd still be able to get the jist of what was going on.

Yes, thank you. That's exactly what I'm driving at here.

At the 'revDB' level things are still too low-level to abstract away database-specific differences

I'm finding that the current structure is too rigid to allow me to integrate a new database type without significantly altering the communication mechanism in a special-case format. And if an overhaul of the current mechanism is in the works, then I'd really like to avoid banging my head against the wall for something that will be obsoleted anyway.

Hi friends, first let me tell you that I agree with the idea: "let MongoDB be MongoDB", said that, do you know about new developments regarding the revDB rework? Regards, Javier