Mailing List Archive: Lucene: Java-Dev

Help: SOLR-3430 and Build changes

Index | Next | Previous | View Threaded

James.Dyer at ingrambook

May 2, 2012, 10:10 AM

Post #1 of 9 (88 views)
Permalink

Help: SOLR-3430 and Build changes

Let me apologize in advance for my (almost) complete ignorance of everything build related: Maven, Ivy, Ant, etc. Sorry!

For Solr-3430, I am introducing a dependency to derby.jar, which will be needed only to run DIH tests. So I don't want it included in the Solr .war. It just needs to be in the classpath when junit runs.

1. Where should I put the .jar/licence/notics/sha1 files?
2. How do I modify the build so that it will be in the classpath for running tests only?
3. What do I need to do to get ivy and maven to pick it up?
4. I'll try my best to get the eclipse/intellij setup correct but I'm only able to test eclipse.

I really want to get this right so please give advice. Thanks.

James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311

rcmuir at gmail

May 2, 2012, 10:21 AM

Post #2 of 9 (80 views)
Permalink

Re: Help: SOLR-3430 and Build changes [In reply to]

On Wed, May 2, 2012 at 1:10 PM, Dyer, James <James.Dyer [at] ingrambook> wrote:
> Let me apologize in advance for my (almost) complete ignorance of everything
> build related: Maven, Ivy, Ant, etc. Sorry!
>
>
>
> For Solr-3430, I am introducing a dependency to derby.jar, which will be
> needed only to run DIH tests. So I don't want it included in the Solr
> .war. It just needs to be in the classpath when junit runs.
>
>
>
> 1. Where should I put the .jar/licence/notics/sha1 files?
>
> 2. How do I modify the build so that it will be in the classpath for running
> tests only?
>

one way to do it is to make a lib-test/ or whatever and put them in
there. Then, you would modify ivy.xml to have two configurations
instead of just the one default, and the additional configuration
would be called 'test' or something like that. Then modify build.xml
to have a different 'resolve' (see solr/example for something that
does just what i describe) and also declare a different test-classpath
that includes lib-test/

that shouldnt be too bad, but the bigger question is with this
test-only dependency, is it really a unit test or more of an
integration test?
For a unit test I think a mock object would be preferred.

If we want to start having integration tests (looking at your issue
title, it really seems thats what you are proposing), then I think it
would be cleaner to just have an integration testing module instead?
Otherwise I can see the tests/dependencies for various modules getting
really complicated quickly, and I worry about wrong things getting
included in packaging or wars or whatever.

As far as having build setup properly for integration tests, I already
proposed such a thing here
https://issues.apache.org/jira/browse/LUCENE-3974

--
lucidimagination.com

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe [at] lucene
For additional commands, e-mail: dev-help [at] lucene

rcmuir at gmail

May 2, 2012, 10:31 AM

Post #3 of 9 (80 views)
Permalink

Re: Help: SOLR-3430 and Build changes [In reply to]

On Wed, May 2, 2012 at 1:10 PM, Dyer, James <James.Dyer [at] ingrambook> wrote:
>
> For Solr-3430, I am introducing a dependency to derby.jar, which will be
> needed only to run DIH tests. So I don't want it included in the Solr
> .war. It just needs to be in the classpath when junit runs.
>

also, on a related note, the example/example-dih currently pulls down
hsqldb.jar.

So another solution could be that your test reuses that jar, or it
could be switched to derby maybe?
solr common-build/build.xml would need to be changed slightly, since
currently this only happens when you do 'ant example'.

Either way it seems best to test the one (hsqldb or derby, i dont
think it matters?) thats being provided in the example i think.

--
lucidimagination.com

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe [at] lucene
For additional commands, e-mail: dev-help [at] lucene

uwe at thetaphi

May 2, 2012, 10:51 AM

Post #4 of 9 (82 views)
Permalink

RE: Help: SOLR-3430 and Build changes [In reply to]

One note:

Derby is included since JDK 6 as "JavaDB" together with the JDK:
http://www.oracle.com/technetwork/java/javadb/overview/index.html

As Lucene/Solr 4 will be using JDK 6 as minimum requirement (in contrast) to Solr 3.x (which was JDK 5), can we not simply rely on this version shipped with JDK? That would make life easy. And for simple tests that version should be enough...

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe [at] thetaphi

> -----Original Message-----
> From: Robert Muir [mailto:rcmuir [at] gmail]
> Sent: Wednesday, May 02, 2012 7:21 PM
> To: dev [at] lucene
> Subject: Re: Help: SOLR-3430 and Build changes
>
> On Wed, May 2, 2012 at 1:10 PM, Dyer, James
> <James.Dyer [at] ingrambook> wrote:
> > Let me apologize in advance for my (almost) complete ignorance of
> > everything build related: Maven, Ivy, Ant, etc. Sorry!
> >
> >
> >
> > For Solr-3430, I am introducing a dependency to derby.jar, which will
> > be needed only to run DIH tests. So I don't want it included in the
> > Solr .war. It just needs to be in the classpath when junit runs.
> >
> >
> >
> > 1. Where should I put the .jar/licence/notics/sha1 files?
> >
> > 2. How do I modify the build so that it will be in the classpath for
> > running tests only?
> >
>
> one way to do it is to make a lib-test/ or whatever and put them in there. Then,
> you would modify ivy.xml to have two configurations instead of just the one
> default, and the additional configuration would be called 'test' or something
> like that. Then modify build.xml to have a different 'resolve' (see solr/example
> for something that does just what i describe) and also declare a different test-
> classpath that includes lib-test/
>
> that shouldnt be too bad, but the bigger question is with this test-only
> dependency, is it really a unit test or more of an integration test?
> For a unit test I think a mock object would be preferred.
>
> If we want to start having integration tests (looking at your issue title, it really
> seems thats what you are proposing), then I think it would be cleaner to just
> have an integration testing module instead?
> Otherwise I can see the tests/dependencies for various modules getting really
> complicated quickly, and I worry about wrong things getting included in
> packaging or wars or whatever.
>
> As far as having build setup properly for integration tests, I already proposed
> such a thing here
> https://issues.apache.org/jira/browse/LUCENE-3974
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe [at] lucene For additional
> commands, e-mail: dev-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe [at] lucene
For additional commands, e-mail: dev-help [at] lucene

rcmuir at gmail

May 2, 2012, 11:42 AM

Post #5 of 9 (81 views)
Permalink

Re: Help: SOLR-3430 and Build changes [In reply to]

On Wed, May 2, 2012 at 1:51 PM, Uwe Schindler <uwe [at] thetaphi> wrote:
> One note:
>
> Derby is included since JDK 6 as "JavaDB" together with the JDK:
> http://www.oracle.com/technetwork/java/javadb/overview/index.html
>
> As Lucene/Solr 4 will be using JDK 6 as minimum requirement (in contrast) to Solr 3.x (which was JDK 5), can we not simply rely on this version shipped with JDK? That would make life easy. And for simple tests that version should be enough...
>

But we dont require *oracle*s implementation as a minimum requirement.
we also support IBM etc too?

--
lucidimagination.com

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe [at] lucene
For additional commands, e-mail: dev-help [at] lucene

uwe at thetaphi

May 2, 2012, 12:16 PM

Post #6 of 9 (80 views)
Permalink

RE: Help: SOLR-3430 and Build changes [In reply to]

I have not checked this, but if the JavaDB is in the JDK official JavaDocs and is therefore part of JDK6 spec? We have to check this, but *if* the package names start with java.db or whatever it *has* to be also in alternate JDK impls. At least OpenJDK also downloads derby while building.

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe [at] thetaphi

> -----Original Message-----
> From: Robert Muir [mailto:rcmuir [at] gmail]
> Sent: Wednesday, May 02, 2012 8:42 PM
> To: dev [at] lucene
> Subject: Re: Help: SOLR-3430 and Build changes
>
> On Wed, May 2, 2012 at 1:51 PM, Uwe Schindler <uwe [at] thetaphi> wrote:
> > One note:
> >
> > Derby is included since JDK 6 as "JavaDB" together with the JDK:
> > http://www.oracle.com/technetwork/java/javadb/overview/index.html
> >
> > As Lucene/Solr 4 will be using JDK 6 as minimum requirement (in contrast) to
> Solr 3.x (which was JDK 5), can we not simply rely on this version shipped with
> JDK? That would make life easy. And for simple tests that version should be
> enough...
> >
>
> But we dont require *oracle*s implementation as a minimum requirement.
> we also support IBM etc too?
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe [at] lucene For additional
> commands, e-mail: dev-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe [at] lucene
For additional commands, e-mail: dev-help [at] lucene

James.Dyer at ingrambook

May 2, 2012, 12:55 PM

Post #7 of 9 (87 views)
Permalink

RE: Help: SOLR-3430 and Build changes [In reply to]

I did a little digging on this and I'm not sure relying on JavaDB is such a sure bet. Its a verbatim copy of Derby 10.2 and while bundled in with the jvm, its not in the classpath by default. Also, I have 2 Oracle 1.6 JVMs on my PC and only 1 includes it. Also, while the documentation says it is in the "db" directory, on my installation its in the "javadb" directory. It would be tricky at best to reliably get this in the tester's classpath, I think. It would be safer I think to just include the jar.

My thoughts were to eventually migrate the example to use derby instead of hsqldb. Maybe I should either change my test to use hsqldb or change the example to use derby. Then as Robert points out, its just a minor build modification to use the jar from the example. In any case, the current Mock datasource doesn't emulate a real JDBC driver very well and I found it was extremely simple to use Derby in in-memory embedded mode (All you do is issue "DriverManager#getConnection" with the correct string). There are no config files, etc.

I don't know if you want to call this a "unit" test or an "integration" test (and what are all those other Solr tests that use Jetty, etc?). In the end, I just want readable tests that are true to real life, which DIH lacks right now.

James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311

-----Original Message-----
From: Uwe Schindler [mailto:uwe [at] thetaphi]
Sent: Wednesday, May 02, 2012 2:16 PM
To: dev [at] lucene
Subject: RE: Help: SOLR-3430 and Build changes

I have not checked this, but if the JavaDB is in the JDK official JavaDocs and is therefore part of JDK6 spec? We have to check this, but *if* the package names start with java.db or whatever it *has* to be also in alternate JDK impls. At least OpenJDK also downloads derby while building.

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe [at] thetaphi

> -----Original Message-----
> From: Robert Muir [mailto:rcmuir [at] gmail]
> Sent: Wednesday, May 02, 2012 8:42 PM
> To: dev [at] lucene
> Subject: Re: Help: SOLR-3430 and Build changes
>
> On Wed, May 2, 2012 at 1:51 PM, Uwe Schindler <uwe [at] thetaphi> wrote:
> > One note:
> >
> > Derby is included since JDK 6 as "JavaDB" together with the JDK:
> > http://www.oracle.com/technetwork/java/javadb/overview/index.html
> >
> > As Lucene/Solr 4 will be using JDK 6 as minimum requirement (in contrast) to
> Solr 3.x (which was JDK 5), can we not simply rely on this version shipped with
> JDK? That would make life easy. And for simple tests that version should be
> enough...
> >
>
> But we dont require *oracle*s implementation as a minimum requirement.
> we also support IBM etc too?
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe [at] lucene For additional
> commands, e-mail: dev-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe [at] lucene
For additional commands, e-mail: dev-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe [at] lucene
For additional commands, e-mail: dev-help [at] lucene

rcmuir at gmail

May 2, 2012, 1:02 PM

Post #8 of 9 (80 views)
Permalink

Re: Help: SOLR-3430 and Build changes [In reply to]

On Wed, May 2, 2012 at 3:55 PM, Dyer, James <James.Dyer [at] ingrambook> wrote:
>
> I don't know if you want to call this a "unit" test or an "integration" test (and what are all those other Solr tests that use Jetty, etc?). In the end, I just want readable tests that are true to real life, which DIH lacks right now.
>

I definitely agree with improving the tests here. I just suggested the
whole issue of integration tests because thats sort of what it is
right? its testing integration with a specific database. You are right
about the Jetty stuff too, when I brought this up recently there seems
to debate over whether solr should work with only jetty or also
others.

But from my perspective tests that are really sucking in the solr
example/ and working against a specific container or database are
really integration tests.

These kind of tests are realistic and useful, I am just trying to
imagine a plan for how such tests are going to work in general,
imagine if you want to test things other than hsqldb in the future...
e.g. at least back in the day doing things like working with blobs in
oracle was... different :)

--
lucidimagination.com

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe [at] lucene
For additional commands, e-mail: dev-help [at] lucene

uwe at thetaphi

May 2, 2012, 1:05 PM

Post #9 of 9 (82 views)
Permalink

RE: Help: SOLR-3430 and Build changes [In reply to]

Hi,

> I did a little digging on this and I'm not sure relying on JavaDB is such a sure
> bet. Its a verbatim copy of Derby 10.2 and while bundled in with the jvm, its
> not in the classpath by default. Also, I have 2 Oracle 1.6 JVMs on my PC and
> only 1 includes it. Also, while the documentation says it is in the "db" directory,
> on my installation its in the "javadb" directory. It would be tricky at best to
> reliably get this in the tester's classpath, I think. It would be safer I think to just
> include the jar.

Thanks for taking care! I just wanted to mention this, as I was browsing Oracles Webpage today (downloading 7u4) and this came into my mind (and I remembered that there was something like javadb, but I was not sure if it was part of official Java spec). So as its only verbatim bundled and uses the apache package name, its definitely not standard, it's just a goodie for JDK. And as it is not even in classpath, forget about it :-)

> My thoughts were to eventually migrate the example to use derby instead of
> hsqldb. Maybe I should either change my test to use hsqldb or change the
> example to use derby. Then as Robert points out, its just a minor build
> modification to use the jar from the example. In any case, the current Mock
> datasource doesn't emulate a real JDBC driver very well and I found it was
> extremely simple to use Derby in in-memory embedded mode (All you do is
> issue "DriverManager#getConnection" with the correct string). There are no
> config files, etc.

I agree here. Emulating a JDBC driver as a Mock is almost undoable, as it would need to implement a lot of interfaces with complex API, including a SQL query parser. That's undoable with few lines of code. Important for JDBC testing is only that you have an in-process database (HSQLDB or as you mention Derby) that needs no disk space and runs completely in memory and needs no configuration. Setting up test cases is easy then, as you say. You just have to create a "Mock" database using some "CREATE TABLE" and "INSERT" statements and you can start testing.

> I don't know if you want to call this a "unit" test or an "integration" test (and
> what are all those other Solr tests that use Jetty, etc?). In the end, I just want
> readable tests that are true to real life, which DIH lacks right now.

I agree!

Uwe

> James Dyer
> E-Commerce Systems
> Ingram Content Group
> (615) 213-4311
>
>
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe [at] thetaphi]
> Sent: Wednesday, May 02, 2012 2:16 PM
> To: dev [at] lucene
> Subject: RE: Help: SOLR-3430 and Build changes
>
> I have not checked this, but if the JavaDB is in the JDK official JavaDocs and is
> therefore part of JDK6 spec? We have to check this, but *if* the package names
> start with java.db or whatever it *has* to be also in alternate JDK impls. At
> least OpenJDK also downloads derby while building.
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe [at] thetaphi
>
>
> > -----Original Message-----
> > From: Robert Muir [mailto:rcmuir [at] gmail]
> > Sent: Wednesday, May 02, 2012 8:42 PM
> > To: dev [at] lucene
> > Subject: Re: Help: SOLR-3430 and Build changes
> >
> > On Wed, May 2, 2012 at 1:51 PM, Uwe Schindler <uwe [at] thetaphi> wrote:
> > > One note:
> > >
> > > Derby is included since JDK 6 as "JavaDB" together with the JDK:
> > > http://www.oracle.com/technetwork/java/javadb/overview/index.html
> > >
> > > As Lucene/Solr 4 will be using JDK 6 as minimum requirement (in
> > > contrast) to
> > Solr 3.x (which was JDK 5), can we not simply rely on this version
> > shipped with JDK? That would make life easy. And for simple tests that
> > version should be enough...
> > >
> >
> > But we dont require *oracle*s implementation as a minimum requirement.
> > we also support IBM etc too?
> >
> > --
> > lucidimagination.com
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe [at] lucene For
> > additional commands, e-mail: dev-help [at] lucene
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe [at] lucene For additional
> commands, e-mail: dev-help [at] lucene
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe [at] lucene For additional
> commands, e-mail: dev-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe [at] lucene
For additional commands, e-mail: dev-help [at] lucene

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads