Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Bricolage: users

New amazon S3 mover

 

 

Bricolage users RSS feed   Index | Next | Previous | View Threaded


adam at RFXTechnologies

Mar 10, 2011, 8:34 AM

Post #1 of 14 (2593 views)
Permalink
New amazon S3 mover

First question, anyone already built an Amazon S3 mover for Bricolage that they are willing to share?

If not, I might give it a whirl. Looks like it is probably way over my head, but I used to say that about Perl, and now I can print a pretty mean "Hello World" page. I see the directions on how to add the mover here -> http://bricolagecms.org/docs/current/api/?Bric::Dist::Action::Mover, is there anywhere else I might look for additional tips and tricks?

Is there also a way to add additional parameters to either the Server Profile or to the Destination Profile? It would be really nice to be able to add a set of properties on the Server Profile page based on the type of Move Method specified in the Destination Profile.
I suppose lacking this, I will also need to figure out how look up values in bricolage.conf.

Adam


david at kineticode

Mar 10, 2011, 9:04 AM

Post #2 of 14 (2524 views)
Permalink
Re: New amazon S3 mover [In reply to]

On Mar 10, 2011, at 8:34 AM, Adam Wilson wrote:

> First question, anyone already built an Amazon S3 mover for Bricolage that they are willing to share?
>
> If not, I might give it a whirl. Looks like it is probably way over my head, but I used to say that about Perl, and now I can print a pretty mean "Hello World" page. I see the directions on how to add the mover here -> http://bricolagecms.org/docs/current/api/?Bric::Dist::Action::Mover, is there anywhere else I might look for additional tips and tricks?

Just look at the code in the other movers.

> Is there also a way to add additional parameters to either the Server Profile or to the Destination Profile? It would be really nice to be able to add a set of properties on the Server Profile page based on the type of Move Method specified in the Destination Profile.
> I suppose lacking this, I will also need to figure out how look up values in bricolage.conf.

Yes. See the email mover.

Best,

David


john.durkin at gmail

Mar 21, 2011, 12:11 PM

Post #3 of 14 (2422 views)
Permalink
Re: New amazon S3 mover [In reply to]

Any progress with this? I'm going to be making one as well. Was trying to
just use the filesystem mover with a fuse mount (s3fs - fuse) - but s3fs
can't handle the way perl is trying to save files (and create directory
structure), so that's out.

best,
John Durkin

On Thu, Mar 10, 2011 at 9:04 AM, David E. Wheeler <david [at] kineticode>wrote:

> On Mar 10, 2011, at 8:34 AM, Adam Wilson wrote:
>
> > First question, anyone already built an Amazon S3 mover for Bricolage
> that they are willing to share?
> >
> > If not, I might give it a whirl. Looks like it is probably way over my
> head, but I used to say that about Perl, and now I can print a pretty mean
> "Hello World" page. I see the directions on how to add the mover here ->
> http://bricolagecms.org/docs/current/api/?Bric::Dist::Action::Mover, is
> there anywhere else I might look for additional tips and tricks?
>
> Just look at the code in the other movers.
>
> > Is there also a way to add additional parameters to either the Server
> Profile or to the Destination Profile? It would be really nice to be able
> to add a set of properties on the Server Profile page based on the type of
> Move Method specified in the Destination Profile.
> > I suppose lacking this, I will also need to figure out how look up values
> in bricolage.conf.
>
> Yes. See the email mover.
>
> Best,
>
> David
>
>
>


david at kineticode

Mar 21, 2011, 2:13 PM

Post #4 of 14 (2423 views)
Permalink
Re: New amazon S3 mover [In reply to]

On Mar 21, 2011, at 12:11 PM, John Durkin wrote:

> Any progress with this? I'm going to be making one as well. Was trying to
> just use the filesystem mover with a fuse mount (s3fs - fuse) - but s3fs
> can't handle the way perl is trying to save files (and create directory
> structure), so that's out.

Something should be do-able with Net::Amazon::S3.

Best,

David


david at kineticode

Mar 21, 2011, 2:13 PM

Post #5 of 14 (2416 views)
Permalink
Re: New amazon S3 mover [In reply to]

On Mar 21, 2011, at 12:11 PM, John Durkin wrote:

> Any progress with this? I'm going to be making one as well. Was trying to
> just use the filesystem mover with a fuse mount (s3fs - fuse) - but s3fs
> can't handle the way perl is trying to save files (and create directory
> structure), so that's out.

Something should be do-able with Net::Amazon::S3.

Best,

David


adam at RFXTechnologies

Mar 22, 2011, 8:23 AM

Post #6 of 14 (2422 views)
Permalink
RE: New amazon S3 mover [In reply to]

For the time being, we are using the filesystem mover with a fuse mount without any issues. I haven't had any issues that I have noticed with this approach, but I only just implemented this a couple of weeks ago, perhaps there are issues I am just unaware of.

I started looking into creating the new mover, but I was immediately overwhelmed by the task at hand. I am planning on using the Net::Amazon::S3,as David stated, but I am not the best perl programmer and it is going to take some time to get anything close to as sophisticated as the existing movers in place. I will keep the thread informed as I make progress though. It will help to motivate me to know that I am not the only one interested in this capability.

Adam

From: John Durkin [mailto:john.durkin [at] gmail]
Sent: Monday, March 21, 2011 3:12 PM
To: users [at] lists
Cc: Adam Wilson
Subject: Re: New amazon S3 mover

Any progress with this? I'm going to be making one as well. Was trying to just use the filesystem mover with a fuse mount (s3fs - fuse) - but s3fs can't handle the way perl is trying to save files (and create directory structure), so that's out.

best,
John Durkin
On Thu, Mar 10, 2011 at 9:04 AM, David E. Wheeler <david [at] kineticode<mailto:david [at] kineticode>> wrote:
On Mar 10, 2011, at 8:34 AM, Adam Wilson wrote:

> First question, anyone already built an Amazon S3 mover for Bricolage that they are willing to share?
>
> If not, I might give it a whirl. Looks like it is probably way over my head, but I used to say that about Perl, and now I can print a pretty mean "Hello World" page. I see the directions on how to add the mover here -> http://bricolagecms.org/docs/current/api/?Bric::Dist::Action::Mover, is there anywhere else I might look for additional tips and tricks?
Just look at the code in the other movers.

> Is there also a way to add additional parameters to either the Server Profile or to the Destination Profile? It would be really nice to be able to add a set of properties on the Server Profile page based on the type of Move Method specified in the Destination Profile.
> I suppose lacking this, I will also need to figure out how look up values in bricolage.conf.
Yes. See the email mover.

Best,

David


john at j2d3

Mar 22, 2011, 9:09 AM

Post #7 of 14 (2416 views)
Permalink
Re: New amazon S3 mover [In reply to]

oh, that's encouraging to hear the fuse mount is working for you... version
of fuse are you using, and on what platform? what version of s3fs?

On Tue, Mar 22, 2011 at 8:23 AM, Adam Wilson <adam [at] rfxtechnologies>wrote:

> For the time being, we are using the filesystem mover with a fuse mount
> without any issues. I havenít had any issues that I have noticed with this
> approach, but I only just implemented this a couple of weeks ago, perhaps
> there are issues I am just unaware of.
>
>
>
> I started looking into creating the new mover, but I was immediately
> overwhelmed by the task at hand. I am planning on using the
> Net::Amazon::S3,as David stated, but I am not the best perl programmer and
> it is going to take some time to get anything close to as sophisticated as
> the existing movers in place. I will keep the thread informed as I make
> progress though. It will help to motivate me to know that I am not the only
> one interested in this capability.
>
>
>
> Adam
>
>
>
> *From:* John Durkin [mailto:john.durkin [at] gmail]
> *Sent:* Monday, March 21, 2011 3:12 PM
> *To:* users [at] lists
> *Cc:* Adam Wilson
> *Subject:* Re: New amazon S3 mover
>
>
>
> Any progress with this? I'm going to be making one as well. Was trying to
> just use the filesystem mover with a fuse mount (s3fs - fuse) - but s3fs
> can't handle the way perl is trying to save files (and create directory
> structure), so that's out.
>
> best,
> John Durkin
>
> On Thu, Mar 10, 2011 at 9:04 AM, David E. Wheeler <david [at] kineticode>
> wrote:
>
> On Mar 10, 2011, at 8:34 AM, Adam Wilson wrote:
>
> > First question, anyone already built an Amazon S3 mover for Bricolage
> that they are willing to share?
> >
> > If not, I might give it a whirl. Looks like it is probably way over my
> head, but I used to say that about Perl, and now I can print a pretty mean
> "Hello World" page. I see the directions on how to add the mover here ->
> http://bricolagecms.org/docs/current/api/?Bric::Dist::Action::Mover, is
> there anywhere else I might look for additional tips and tricks?
>
> Just look at the code in the other movers.
>
>
> > Is there also a way to add additional parameters to either the Server
> Profile or to the Destination Profile? It would be really nice to be able
> to add a set of properties on the Server Profile page based on the type of
> Move Method specified in the Destination Profile.
> > I suppose lacking this, I will also need to figure out how look up values
> in bricolage.conf.
>
> Yes. See the email mover.
>
> Best,
>
> David
>
>
>


adam at RFXTechnologies

Mar 22, 2011, 1:02 PM

Post #8 of 14 (2438 views)
Permalink
RE: New amazon S3 mover [In reply to]

I used this page heavily for tips on getting it all working: http://code.google.com/p/s3fs/wiki/FuseOverAmazon

Linux 2.6.26-2-686 #1 SMP Thu Sep 16 19:35:51 UTC 2010 i686 GNU/Linux
Fuse: libfuse-dev_2.8.4-1.1_i386, libfuse2_2.8.4-1.1_i386
S3FS: s3fs-1.40

My fstab looks like this:
s3fs#shared.rfxwebsites.com /mnt/s3fs/shared.rfxwebsites.com fuse use_rrs=1,allow_other,nonempty,uid=www-data,gid=www-data,rw,auto,default_acl=public-read 0 0

I set up an additional mount for each website that we host that is related to a bucket on Amazon's S3 architecture.

Let me know if you have any more questions, I am happy to share my limited knowledge when I can.
Adam

From: John Durkin [mailto:john [at] j2d3]
Sent: Tuesday, March 22, 2011 12:09 PM
To: Adam Wilson
Cc: users [at] lists
Subject: Re: New amazon S3 mover

oh, that's encouraging to hear the fuse mount is working for you... version of fuse are you using, and on what platform? what version of s3fs?
On Tue, Mar 22, 2011 at 8:23 AM, Adam Wilson <adam [at] rfxtechnologies<mailto:adam [at] rfxtechnologies>> wrote:
For the time being, we are using the filesystem mover with a fuse mount without any issues. I haven't had any issues that I have noticed with this approach, but I only just implemented this a couple of weeks ago, perhaps there are issues I am just unaware of.

I started looking into creating the new mover, but I was immediately overwhelmed by the task at hand. I am planning on using the Net::Amazon::S3,as David stated, but I am not the best perl programmer and it is going to take some time to get anything close to as sophisticated as the existing movers in place. I will keep the thread informed as I make progress though. It will help to motivate me to know that I am not the only one interested in this capability.

Adam

From: John Durkin [mailto:john.durkin [at] gmail<mailto:john.durkin [at] gmail>]
Sent: Monday, March 21, 2011 3:12 PM
To: users [at] lists<mailto:users [at] lists>
Cc: Adam Wilson
Subject: Re: New amazon S3 mover

Any progress with this? I'm going to be making one as well. Was trying to just use the filesystem mover with a fuse mount (s3fs - fuse) - but s3fs can't handle the way perl is trying to save files (and create directory structure), so that's out.

best,
John Durkin
On Thu, Mar 10, 2011 at 9:04 AM, David E. Wheeler <david [at] kineticode<mailto:david [at] kineticode>> wrote:
On Mar 10, 2011, at 8:34 AM, Adam Wilson wrote:

> First question, anyone already built an Amazon S3 mover for Bricolage that they are willing to share?
>
> If not, I might give it a whirl. Looks like it is probably way over my head, but I used to say that about Perl, and now I can print a pretty mean "Hello World" page. I see the directions on how to add the mover here -> http://bricolagecms.org/docs/current/api/?Bric::Dist::Action::Mover, is there anywhere else I might look for additional tips and tricks?
Just look at the code in the other movers.

> Is there also a way to add additional parameters to either the Server Profile or to the Destination Profile? It would be really nice to be able to add a set of properties on the Server Profile page based on the type of Move Method specified in the Destination Profile.
> I suppose lacking this, I will also need to figure out how look up values in bricolage.conf.
Yes. See the email mover.

Best,

David


david at kineticode

Mar 23, 2011, 9:14 AM

Post #9 of 14 (2410 views)
Permalink
Re: New amazon S3 mover [In reply to]

On Mar 22, 2011, at 11:23 AM, Adam Wilson wrote:

> For the time being, we are using the filesystem mover with a fuse mount without any issues. I haven't had any issues that I have noticed with this approach, but I only just implemented this a couple of weeks ago, perhaps there are issues I am just unaware of.

Seems like a perfectly reasonable approach.

> I started looking into creating the new mover, but I was immediately overwhelmed by the task at hand. I am planning on using the Net::Amazon::S3,as David stated, but I am not the best perl programmer and it is going to take some time to get anything close to as sophisticated as the existing movers in place. I will keep the thread informed as I make progress though. It will help to motivate me to know that I am not the only one interested in this capability.

Thanks, but if you have something working, there's obviously no rush. Take your time, we can help you to get it right.

Best,

David


david at kineticode

Mar 23, 2011, 9:14 AM

Post #10 of 14 (2411 views)
Permalink
Re: New amazon S3 mover [In reply to]

On Mar 22, 2011, at 11:23 AM, Adam Wilson wrote:

> For the time being, we are using the filesystem mover with a fuse mount without any issues. I haven't had any issues that I have noticed with this approach, but I only just implemented this a couple of weeks ago, perhaps there are issues I am just unaware of.

Seems like a perfectly reasonable approach.

> I started looking into creating the new mover, but I was immediately overwhelmed by the task at hand. I am planning on using the Net::Amazon::S3,as David stated, but I am not the best perl programmer and it is going to take some time to get anything close to as sophisticated as the existing movers in place. I will keep the thread informed as I make progress though. It will help to motivate me to know that I am not the only one interested in this capability.

Thanks, but if you have something working, there's obviously no rush. Take your time, we can help you to get it right.

Best,

David


adam at RFXTechnologies

Jul 13, 2011, 8:02 AM

Post #11 of 14 (2094 views)
Permalink
RE: New amazon S3 mover [In reply to]

John, et. Al.,
I am *really* going to be doing this today, unless it has already been done.
2 questions, the first is please let me know if someone has already built this.
The second question, is there somewhere that features are being tracked in github? I see 27 issues there(seems like a small number) but not sure that features should go there. But if not there, then how do we know what features are out there and coordinate working on them?

Thanks for the advice,
Adam

From: John Durkin [mailto:john.durkin [at] gmail]
Sent: Monday, March 21, 2011 3:12 PM
To: users [at] lists
Cc: Adam Wilson
Subject: Re: New amazon S3 mover

Any progress with this? I'm going to be making one as well. Was trying to just use the filesystem mover with a fuse mount (s3fs - fuse) - but s3fs can't handle the way perl is trying to save files (and create directory structure), so that's out.

best,
John Durkin
On Thu, Mar 10, 2011 at 9:04 AM, David E. Wheeler <david [at] kineticode<mailto:david [at] kineticode>> wrote:
On Mar 10, 2011, at 8:34 AM, Adam Wilson wrote:

> First question, anyone already built an Amazon S3 mover for Bricolage that they are willing to share?
>
> If not, I might give it a whirl. Looks like it is probably way over my head, but I used to say that about Perl, and now I can print a pretty mean "Hello World" page. I see the directions on how to add the mover here -> http://bricolagecms.org/docs/current/api/?Bric::Dist::Action::Mover, is there anywhere else I might look for additional tips and tricks?
Just look at the code in the other movers.

> Is there also a way to add additional parameters to either the Server Profile or to the Destination Profile? It would be really nice to be able to add a set of properties on the Server Profile page based on the type of Move Method specified in the Destination Profile.
> I suppose lacking this, I will also need to figure out how look up values in bricolage.conf.
Yes. See the email mover.

Best,

David


david at kineticode

Jul 13, 2011, 9:13 AM

Post #12 of 14 (2091 views)
Permalink
Re: New amazon S3 mover [In reply to]

On Jul 13, 2011, at 8:02 AM, Adam Wilson wrote:

> John, et. Al.,
> I am *really* going to be doing this today, unless it has already been done.
> 2 questions, the first is please let me know if someone has already built this.

Not to my knowledge.

> The second question, is there somewhere that features are being tracked in github? I see 27 issues there(seems like a small number) but not sure that features should go there. But if not there, then how do we know what features are out there and coordinate working on them?

Lighthouse.

http://bricolage.lighthouseapp.com/

I don't see any open issues in the GitHub tracker, just a single pull request.

https://github.com/bricoleurs/bricolage/issues

That's how we coordinate pull requests, of course. See the "Getting the Source" section of https://github.com/bricoleurs/bricolage/wiki.

HTH,

David


david at kineticode

Jul 13, 2011, 9:13 AM

Post #13 of 14 (2094 views)
Permalink
Re: New amazon S3 mover [In reply to]

On Jul 13, 2011, at 8:02 AM, Adam Wilson wrote:

> John, et. Al.,
> I am *really* going to be doing this today, unless it has already been done.
> 2 questions, the first is please let me know if someone has already built this.

Not to my knowledge.

> The second question, is there somewhere that features are being tracked in github? I see 27 issues there(seems like a small number) but not sure that features should go there. But if not there, then how do we know what features are out there and coordinate working on them?

Lighthouse.

http://bricolage.lighthouseapp.com/

I don't see any open issues in the GitHub tracker, just a single pull request.

https://github.com/bricoleurs/bricolage/issues

That's how we coordinate pull requests, of course. See the "Getting the Source" section of https://github.com/bricoleurs/bricolage/wiki.

HTH,

David


john at j2d3

Jul 13, 2011, 10:28 AM

Post #14 of 14 (2091 views)
Permalink
Re: New amazon S3 mover [In reply to]

I built this - I have an EC2 image with a successful implementation of my S3
Mover... I did 99% of it - that last 1% - sharing it with anyone - that's
the part i gotta still do.

So - I'm spinning up that instance again (I'd stopped it, was not in use) -
I will refresh my memory as to the files I changed...

I am super busy, so Adam - if you contact me separately maybe we can figure
out a way for you to login to that instance and do the last 1% of the work?

JD

On Wed, Jul 13, 2011 at 9:13 AM, David E. Wheeler <david [at] kineticode>wrote:

> On Jul 13, 2011, at 8:02 AM, Adam Wilson wrote:
>
> > John, et. Al.,
> > I am *really* going to be doing this today, unless it has
> already been done.
> > 2 questions, the first is please let me know if someone
> has already built this.
>
> Not to my knowledge.
>
> > The second question, is there somewhere that features are
> being tracked in github? I see 27 issues there(seems like a small number)
> but not sure that features should go there. But if not there, then how do
> we know what features are out there and coordinate working on them?
>
> Lighthouse.
>
> http://bricolage.lighthouseapp.com/
>
> I don't see any open issues in the GitHub tracker, just a single pull
> request.
>
> https://github.com/bricoleurs/bricolage/issues
>
> That's how we coordinate pull requests, of course. See the "Getting the
> Source" section of https://github.com/bricoleurs/bricolage/wiki.
>
> HTH,
>
> David
>
>

Bricolage users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.