Mailing List Archive: Catalyst: Users

KiokuDB, MongoDB and the NoSQL thing

Index | Next | Previous | View Threaded

keenlinks at ymail

Mar 1, 2010, 10:07 PM

Post #1 of 11 (2406 views)
Permalink

KiokuDB, MongoDB and the NoSQL thing

I have a couple of production Catalyst/DBIx::Class sites on Debian stable, and then on my personal hobby site I use local::lib to try out new things. Recently I split out my users for this site into a separate model and I thought it a good exercise to learn and use KiokuDB. It was just a couple of simple objects, users & roles, but I believe I have a better understanding of how a schema-less data model would work. All I do are lookups based on ID or indexed object values, but doing any type of ordering by dates or titles is a mystery. It seems that the Search::GIN is to provide this sort of functionality, but it is under-documented and has not had an update for awhile.

I do
not see many posts
regarding uses of KiokuDB within Catalyst so I was curious about the opinion of the community in regards to its usage. Is it still to early within development?

Also, I have been reading more about the increase in the NoSQL interest, with a particular interest in the MongoDB database (it seems to be similar in some respects to KiokuDB), but I do not find Perl people in the discussion as much as others (Ruby, PHP). Are there developers in the Catalyst community who lean toward NoSQL concepts over traditional RDMS's, or is best to view as a tool to use at times?

How about MongoDB? Am I being suckered by another bandwagon?

Thanks, Scott K.

ido at ido50

Mar 1, 2010, 11:06 PM

Post #2 of 11 (2353 views)
Permalink

Re: KiokuDB, MongoDB and the NoSQL thing [In reply to]

This is actually a pretty interesting subject, and while I can't help you
with any of it as I have no experience (I have tried KiokuDB once for a
project I was working on, but found that it's too young for my purposes,
from the same reason as you - searching and sorting is next to impossible).

Anyway, I hope this subject gets more attention, as the possibilities are
huge.

Ido.

On Tue, Mar 2, 2010 at 8:07 AM, S.A. Kiehn <keenlinks [at] ymail> wrote:

> I have a couple of production Catalyst/DBIx::Class sites on Debian stable,
> and then on my personal hobby site I use local::lib to try out new things.
> Recently I split out my users for this site into a separate model and I
> thought it a good exercise to learn and use KiokuDB. It was just a couple
> of simple objects, users & roles, but I believe I have a better
> understanding of how a schema-less data model would work. All I do are
> lookups based on ID or indexed object values, but doing any type of ordering
> by dates or titles is a mystery. It seems that the Search::GIN is to
> provide this sort of functionality, but it is under-documented and has not
> had an update for awhile.
>
> I do not see many posts regarding uses of KiokuDB within Catalyst so I was
> curious about the opinion of the community in regards to its usage. Is it
> still to early within development?
>
> Also, I have been reading more about the increase in the NoSQL interest,
> with a particular interest in the MongoDB database (it seems to be similar
> in some respects to KiokuDB), but I do not find Perl people in the
> discussion as much as others (Ruby, PHP). Are there developers in the
> Catalyst community who lean toward NoSQL concepts over traditional RDMS's,
> or is best to view as a tool to use at times?
>
> How about MongoDB? Am I being suckered by another bandwagon?
>
> Thanks, Scott K.
>
>
>
>
>
>
>
>
> _______________________________________________
> List: Catalyst [at] lists
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst [at] lists/
> Dev site: http://dev.catalyst.perl.org/
>
>

toby.corkindale at strategicdata

Mar 1, 2010, 11:31 PM

Post #3 of 11 (2336 views)
Permalink

Re: KiokuDB, MongoDB and the NoSQL thing [In reply to]

On 02/03/10 17:07, S.A. Kiehn wrote:
> I have a couple of production Catalyst/DBIx::Class sites on Debian
> stable, and then on my personal hobby site I use local::lib to try out
> new things. Recently I split out my users for this site into a separate
> model and I thought it a good exercise to learn and use KiokuDB. It was
> just a couple of simple objects, users & roles, but I believe I have a
> better understanding of how a schema-less data model would work. All I
> do are lookups based on ID or indexed object values, but doing any type
> of ordering by dates or titles is a mystery. It seems that the
> Search::GIN is to provide this sort of functionality, but it is
> under-documented and has not had an update for awhile.
>
> I do not see many posts regarding uses of KiokuDB within Catalyst so I
> was curious about the opinion of the community in regards to its usage.
> Is it still to early within development?
>
> Also, I have been reading more about the increase in the NoSQL interest,
> with a particular interest in the MongoDB database (it seems to be
> similar in some respects to KiokuDB), but I do not find Perl people in
> the discussion as much as others (Ruby, PHP). Are there developers in
> the Catalyst community who lean toward NoSQL concepts over traditional
> RDMS's, or is best to view as a tool to use at times?
>
> How about MongoDB? Am I being suckered by another bandwagon?

Also Apache CouchDB.

I'm curious to know how these things work out for what I see as "real
world" cases, where you do actually want to do searches on correlated data.

For instance, say you wanted to make a phpbb-style forum.
You have several forum areas, and within each there are many threads,
each containing many posts.

Would every post in a thread be a new document, or would the entire
thread be one big document? How would they be linked to the forums, by
an ID in the document, or do we get into some kind of mega-document
encapsulating everything on the board?

Say I wanted to do a search for all posts made by users named John in
the forum called Linux, what would the syntax look like?

cheers,
Toby

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

darren at darrenduncan

Mar 2, 2010, 12:30 AM

Post #4 of 11 (2349 views)
Permalink

Re: KiokuDB, MongoDB and the NoSQL thing [In reply to]

S.A. Kiehn wrote:
> I do not see many posts regarding uses of KiokuDB within Catalyst so I
> was curious about the opinion of the community in regards to its usage.
> Is it still to early within development?
>
> Also, I have been reading more about the increase in the NoSQL interest,
> with a particular interest in the MongoDB database (it seems to be
> similar in some respects to KiokuDB), but I do not find Perl people in
> the discussion as much as others (Ruby, PHP). Are there developers in
> the Catalyst community who lean toward NoSQL concepts over traditional
> RDMS's, or is best to view as a tool to use at times?
>
> How about MongoDB? Am I being suckered by another bandwagon?
>
> Thanks, Scott K.

Well I happen to be strongly opinionated on this topic, so here goes ...

While these other DBMSs have their uses, I believe that anyone is misguided who
figures they are superior solutions for most uses of relational databases.

I believe that the relational model of data is still the single best general
solution that we have come up with for organizing and querying any non-trivial
amount of data that is the least bit structured. Especially so when that data
needs to be or could possibly be worked with by multiple kinds of users or
applications that may have different needs, and need their own views of that
data. It is also good for persisting data, but persistence isn't the main
point; rather easy and flexible organization and querying is the point;
persistence is optional, same as persisting an array of data is optional.

That said, while it does the job well enough many times, SQL is deeply flawed
and doesn't represent the relational model of data properly, but just
approximates it to varying degrees, this variance depends partly on the SQL DBMS
in use, which range in features quite a bit.

I believe that quite often when people complain about "relational databases",
they are really complaining about "SQL databases", as if those were the same,
and so various "solutions" are proposed like ORMs or these NoSQL concepts.

But the problem is that they are throwing out the baby with the bathwater. Yes,
SQL is quite flawed, but the relational model it approximates is not (or it is
much less flawed, if you want to argue that having something actually flawless
is impossible).

I believe that the best solution is not to ditch everything, but rather to
provide a DBMS that actually implements the full relational model of data, and
not just an approximation, nor ditching the concept entirely. If you do that,
then a lot of these other kinds of DBMSs like so-called "object-relational" or
"object" or "key-value" become redundant, because the full relational model
incorporates their features.

For example, the full relational model supports having tuple/row and
relation/table values as attributes/fields of other ones, and so you can
natively model any arbitrarily complex data type that an object could model,
without too much indirection (similarly to how many languages support having an
array as an element of an array). Hence ORMs are redundant and so are distinct
concepts of object stores, or alternately they become a lot thinner.

And so, as a model of good community behaviour, I'm not just sitting around
talking about what people should do, but I am going out and actually doing it,
creating a DBMS that provides the full relational model (both as self-contained
implementations as well as implementations over existing DBMSs), and right now.
Simultaneously taking what SQL should have been and all the good features of
the SQL alternatives, elegantly integrated in one streamlined bloat-free package.

This project, with the umbrella name "Muldis" (see CPAN etc), is mostly
pre-alpha right now, but I am filling in the gaps as soon as I can and I am
confident it will be viable. In fact, I released updates to 3 of 4 current
sub-projects on CPAN earlier today.

So, to answer your question, go ahead and explore the alternatives you name, but
I will say to anyone that the relational model is the single best general
solution for data aggregation and processing.

-- Darren Duncan

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

hakim.cassimally at gmail

Mar 2, 2010, 2:27 AM

Post #5 of 11 (2359 views)
Permalink

Re: KiokuDB, MongoDB and the NoSQL thing [In reply to]

On 2 March 2010 08:30, Darren Duncan <darren [at] darrenduncan> wrote:
> S.A. Kiehn wrote:
>>
>> I do not see many posts regarding uses of KiokuDB within Catalyst so I was
>> curious about the opinion of the community in regards to its usage. Is it
>> still to early within development?
>>
> Well I happen to be strongly opinionated on this topic, so here goes ...
>
> While these other DBMSs have their uses, I believe that anyone is misguided
> who figures they are superior solutions for most uses of relational
> databases.

I think you may be right. However I've spent barely 3 or 4 hours
looking at how to port an existing application to Moose/KiokuDB and I
think it may be an interesting thing to do. Our current logic uses DB
tables, with DBIx::Class, and some shims like ::DynamicSubclass. It's
powerful, flexible, fast, and very sophisticated for reporting.

But really, the more I think about it, our business objects would
benefit from living in a treelike structure: a "module" might contain
submodules with different business rules etc.. We can cope with the
existing logic, but the roadmap of new features contains a few things
that would seem to require massive re-engineering of our tables...

... and though I'm sure all this business logic /can/ be represented
in a relational DB, it might just require some deep thought.
With Moose, you just create your objects and the rules that tie them
together; then you plug it into KiokuDB and job done.
And when you change your object structure, you don't need to think
about how to model it, or how that will affect various other
relationships and queries. I added the bulk of one of the
"complicated" features I was worrying about representing with DBIC in
around 30 minutes.

(I'm sure I'm missing various frustrations and there may be pitfalls.
On the other hand Yuval and co at iinteractive are Very Clever (TM) so
perhaps I'm not?)

So my first impression (after a few hours) is that

* using Moose + KiokuDB will be a fantastic way to rapid-prototype
some complex business requirements

* I'll hold judgement on how well it scales until I've done a test
with a large sample database. I suspect that starting from known
nodes and walking the tree to data I need will be reasonably fast.
(That's the major use case). I know KiokuDB is reported to be "fast
enough" for small-to-medium datasets, I don't know if anyone's got
good benchmarks from large-to stupidly-large datasets, but it would be
reassuring to have such data!

* On the other hand, I suspect that reporting/random access search
will be slower. That's a lesser use case, but still important. But
when the object structure is solidified enough to know *what* I need
to search/report on, then I can add some stuff to generate some
summary tables etc. to query with good ol' DBI

--
osf'

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

andrew at cleverdomain

Mar 2, 2010, 2:40 AM

Post #6 of 11 (2336 views)
Permalink

Re: KiokuDB, MongoDB and the NoSQL thing [In reply to]

On Tuesday 02 March 2010 12:07:23 am S.A. Kiehn wrote:
> I have a couple of production Catalyst/DBIx::Class sites on Debian stable,
> and then on my personal hobby site I use local::lib to try out new
> things. Recently I split out my users for this site into a separate model
> and I thought it a good exercise to learn and use KiokuDB. It was just a
> couple of simple objects, users & roles, but I believe I have a better
> understanding of how a schema-less data model would work. All I do are
> lookups based on ID or indexed object values, but doing any type of
> ordering by dates or titles is a mystery. It seems that the Search::GIN
> is to provide this sort of functionality, but it is under-documented and
> has not had an update for awhile.
>
> I do
> not see many posts
> regarding uses of KiokuDB within Catalyst so I was curious about the
> opinion of the community in regards to its usage. Is it still to early
> within development?
>
> Also, I have been reading more about the increase in the NoSQL interest,
> with a particular interest in the MongoDB database (it seems to be similar
> in some respects to KiokuDB), but I do not find Perl people in the
> discussion as much as others (Ruby, PHP). Are there developers in the
> Catalyst community who lean toward NoSQL concepts over traditional RDMS's,
> or is best to view as a tool to use at times?
>
> How about MongoDB? Am I being suckered by another bandwagon?
>
Kioku is a really beautiful idea. I was hoping to use it on a recent project
but unfortunately it wasn't the right fit -- one of the most common access
patterns I would have to support for this app was unduly difficult in Kioku at
this time so I decided to put it off. So, no real first-hand advice from me.

Might I suggest hitting #catalyst and #kiokudb on irc.perl.org? I know that
there's at least one person using MongoDB with Catalyst on #catalyst, so you
can probably get some words of wisdom.

Andrew

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

darren at darrenduncan

Mar 2, 2010, 11:07 PM

Post #7 of 11 (2299 views)
Permalink

Re: KiokuDB, MongoDB and the NoSQL thing [In reply to]

Hakim Cassimally wrote:
> I think you may be right. However I've spent barely 3 or 4 hours
> looking at how to port an existing application to Moose/KiokuDB and I
> think it may be an interesting thing to do. Our current logic uses DB
> tables, with DBIx::Class, and some shims like ::DynamicSubclass. It's
> powerful, flexible, fast, and very sophisticated for reporting.
>
> But really, the more I think about it, our business objects would
> benefit from living in a treelike structure: a "module" might contain
> submodules with different business rules etc.. We can cope with the
> existing logic, but the roadmap of new features contains a few things
> that would seem to require massive re-engineering of our tables...
>
> ... and though I'm sure all this business logic /can/ be represented
> in a relational DB, it might just require some deep thought.
> With Moose, you just create your objects and the rules that tie them
> together; then you plug it into KiokuDB and job done.
> And when you change your object structure, you don't need to think
> about how to model it, or how that will affect various other
> relationships and queries. I added the bulk of one of the
> "complicated" features I was worrying about representing with DBIC in
> around 30 minutes.

At any time in the near future, I would be happy if you, or anyone, wanted to
share with me (most likely privately, and I can sign NDAs if needed) their
particular business rules, ideally in the form of a requirements document that
just says what you actually want to express in human terms, rather than saying
it in the form of SQL or Perl classes because it is very likely that those would
contain significant circumlocutions (and hence would add details that are
actually unimportant but I can't tell), and then I will try to formulate Muldis
D renditions of your business rules, and see if Muldis D can represent them more
faithfully or with less circumlocution than other methods, and if not then this
may point out things to improve in Muldis D's design; its fine to include the
SQL or Perl also as illustration, but separate requirements are important for
best interpretation of those; thank you. -- Darren Duncan

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

darren at darrenduncan

Mar 4, 2010, 3:38 PM

Post #8 of 11 (2138 views)
Permalink

Re: KiokuDB, MongoDB and the NoSQL thing [In reply to]

Here's something timely on Ars, in that I just discovered it now around the time
of this discussion thread:

http://arstechnica.com/business/data-centers/2010/02/-since-the-rise-of.ars

It lays out a summary of how the SQL and NoSQL worlds compare, and what sort of
trade-offs you get for each choice, and it introduces several specific NoSQL
projects. I also learned something, such that what CouchDB and MongoDB
specifically represent data with is JSON documents.

So basically, what you pick out of today's choices depends on your priorities.

-- Darren Duncan

S.A. Kiehn wrote:
> I have a couple of production Catalyst/DBIx::Class sites on Debian
> stable, and then on my personal hobby site I use local::lib to try out
> new things. Recently I split out my users for this site into a separate
> model and I thought it a good exercise to learn and use KiokuDB. It was
> just a couple of simple objects, users & roles, but I believe I have a
> better understanding of how a schema-less data model would work. All I
> do are lookups based on ID or indexed object values, but doing any type
> of ordering by dates or titles is a mystery. It seems that the
> Search::GIN is to provide this sort of functionality, but it is
> under-documented and has not had an update for awhile.
>
> I do not see many posts regarding uses of KiokuDB within Catalyst so I
> was curious about the opinion of the community in regards to its usage.
> Is it still to early within development?
>
> Also, I have been reading more about the increase in the NoSQL interest,
> with a particular interest in the MongoDB database (it seems to be
> similar in some respects to KiokuDB), but I do not find Perl people in
> the discussion as much as others (Ruby, PHP). Are there developers in
> the Catalyst community who lean toward NoSQL concepts over traditional
> RDMS's, or is best to view as a tool to use at times?
>
> How about MongoDB? Am I being suckered by another bandwagon?
>
> Thanks, Scott K.

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

ash_cpan at firemirror

Mar 4, 2010, 3:43 PM

Post #9 of 11 (2136 views)
Permalink

Re: KiokuDB, MongoDB and the NoSQL thing [In reply to]

On 4 Mar 2010, at 23:38, Darren Duncan wrote:
> MongoDB specifically represent data with is JSON documents.

Not quite true. They are BSON (B = binary) and have specific types for dates, regexps and binary data

keenlinks at ymail

Mar 5, 2010, 12:59 PM

Post #10 of 11 (2076 views)
Permalink

Re: KiokuDB, MongoDB and the NoSQL thing [In reply to]

Thanks for the link Darren. I will read and ponder.

Also of interest, I seen that Search::GIN has had a new release. The post also mentions some docs explaining queries, but I have not found them yet. Active Search::GIN development would encourage KiokuDB usage I would think.

Thanks for the posts regarding this topic.

S. Kiehn

Darren Duncan wrote:

Here's something timely on Ars, in that I just discovered it now around the time of this discussion thread:

http://arstechnica.com/business/data-centers/2010/02/-since-the-rise-of.ars

It lays out a summary of how the SQL and NoSQL worlds compare, and what sort of trade-offs you get for each choice, and it introduces several specific NoSQL projects. I also learned something, such that what CouchDB and MongoDB specifically represent data with is JSON documents.

So basically, what you pick out of today's choices depends on your priorities.

darren at darrenduncan

Mar 20, 2010, 12:34 AM

Post #11 of 11 (1568 views)
Permalink

Re: KiokuDB, MongoDB and the NoSQL thing [In reply to]

As an addendum to this thread ...

It just occurred to me now, following an interview today with a developer of
millions-users social-network games, that I could be embracing this NoSQL thing
in an indirect way, taking into account that still I consider it just a
reasonable compromise for some applications, but not a general database pancea.

Because Muldis D is designed from the start to be implementable over both SQL
DBMSs of various capabilities and within general purpose languages, meaning that
any functionality a DBMS doesn't natively support can be hoisted to the
application, this structure is a natural fit for NoSQL.

NoSQL is all about keeping the main DBMS simple, supporting just a few native
operations, and massively parallel. And so, one could write Muldis D code and
deploy it both over SQL and NoSQL, and in the latter case it would just do a lot
of the work on the application side. Essentially, Muldis D takes care of the "I
believe I did, Bob", so you can use a NoSQL store without doing the "write a
distributed map-reduce function in Erlang" yourself, instead you just say what
you want declaratively much like in SQL, and then the right thing happens.

Of course, that won't generally give you the consistency/integrity that a
relational DBMS backend has over a NoSQL one, but at least you'll get the
powerful queries part.

You heard it here first.

-- Darren Duncan

S.A. Kiehn wrote:
> Thanks for the link Darren. I will read and ponder.
>
> Also of interest, I seen that Search::GIN has had a new release
> <http://blogs.perl.org/users/sawyer_x/2010/03/searchgin-004-finally-out.html>.
> The post also mentions some docs explaining queries, but I have not
> found them yet. Active Search::GIN development would encourage KiokuDB
> usage I would think.
>
> Thanks for the posts regarding this topic.
>
> S. Kiehn
>
> Darren Duncan wrote:
>
> Here's something timely on Ars, in that I just discovered it now
> around the time of this discussion thread:
>
> http://arstechnica.com/business/data-centers/2010/02/-since-the-rise-of.ars
>
> It lays out a summary of how the SQL and NoSQL worlds compare, and
> what sort of trade-offs you get for each choice, and it introduces
> several specific NoSQL projects. I also learned something, such
> that what CouchDB and MongoDB specifically represent data with is
> JSON documents.
>
> So basically, what you pick out of today's choices depends on your
> priorities.

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads