Mailing List Archive: Bricolage: users

Blogger to Bricolage 2.0 migration

Index | Next | Previous | View Threaded

mattrolf at me

Oct 12, 2010, 7:07 PM

Post #1 of 6 (758 views)
Permalink

Blogger to Bricolage 2.0 migration

Over the last few months I migrated a site I do on the side from Blogger to Bricolage 2.0. In addition to the migration, I did a full site redesign, which you can read about below if things like that interest you. The topic of the site is American soccer.

http://soccer.fakesigi.com/
http://soccer.fakesigi.com/about_fake_sigi_website_redesign.html

I also built the templates from scratch, and when I get a few moments, I'm going to contribute them back to the community with instructions on how to use them. I think they would be very good for a "beginner" site, in that they can build regular web pages, blog posts, keyword archives, a sitemap, and an Atom feed. If you look at the source code, you'll see the HTML is very clean and concise. The templates are the same, and should be understandable for newbies.

First, what's it like running Bric 2? Much nicer than Bric 1 in many ways. The install is much easier. The new desk interface and content editing interface are very much improved over 1.10. The new element constraints are a great feature. Apache 2 and Modperl 2 usage. I have not tried Mysql, so I can not report on that. But from an editor's perspective, this is a much friendlier release. Also, the new awareness in the burner to not republish repeated pages is fantastic.

As for the bad - I am currently using JS-Quicktags to get around issues with the Xinha editor. Some of you may not find that disappointing, others will retch. Document publication is still a bottle neck, as I can burn around 800 pages in 5 minutes, only to wait 20-25 minutes for those to publish out over SFTP. I had to apply some patches that are not in 2.0 yet to get SFTP publishing to work, with the gracious advice of my former colleagues at Denison, and David actually has rewritten the framework from the ground up for the next release. Also, I have been unable to get any other browser other than Firefox to work, as I've documented on the list. I've found a few other minor bugs which will hopefully get patched before the next release.

The migration itself was not bad. I got the XML dump of my blog from Blogger, and used a perl script with XML::XSLT and an XSL Template to convert the Blogger XML into something SOAP-friendly. However, I also had to run some regex to clean up some of the code. In particular, google has taken to dumping a unique ID into the content field of their blogs, which was a little tricky to get out.

Once I had the SOAP friendly XML, I imported the content without much issue. I then set about tweaking the design, implementing a KinoSearch search engine, and doing a whole bunch of keyword edits. I worked on the transition from May through August, and Bricolage work (install, tuning, document modeling, templating, permissions) took probably only a quarter of the time. My goal was to keep it very simple.

Probably the most interesting thing I did template wise was with the keywords. For any keyword added to a story, on publish I look for a keyword archive story with a corresponding title and publish that as well. Keyword stories must be added manually, so keywords can not be added on the fly when editing a story. I knew Phillip has some template code to automate this, but as I mentioned to David, I think Keyword archive-story management code built-in to the system would be really useful.

For hardware I'm running two machines. The Postgresql database is on a 1.6Ghz Core 2 MacMini with a 250GB HD and 2GB RAM. Bricolage is on a 2.0Ghz single-core Athlon from early 2005, with a 80GB HD and 1GB RAM. This setup works very well, as giving Postgres it's own machine does wonders in freeing up Bricolage from a lot of I/O issues. The MacMini never registers over 30% processor load in top, and is probably HD-bound. The Bricolage machine could benefit from another HD or disk array for burning and moving pages, and a single core would not be enough for a serious shop, as apache and bric_queued start competing for resources. Again, SFTP publishing is far and away the biggest performance bottleneck.

The migration was about 800 stories - I did not do any media. It was very simple, and I kept the feature list down so as to not complicate it. Once the new site was up and beta tested for about three weeks, migrating was as simple as redirecting the DNS to the new site. There were no issues upon startup, and I've just had to tweak a few CSS rules.

Overall, I'm very pleased with Bric 2. Let me know if you have questions or if I left anything out here. I'm up for Bric 2 migration consulting if anyone has a gig. ;)

-Matt

phillip at communitybandwidth

Oct 13, 2010, 7:26 AM

Post #2 of 6 (727 views)
Permalink

Re: Blogger to Bricolage 2.0 migration [In reply to]

Matt,

That site is blazingly fast. Nicely done. :) And many thanks for the great writeup. Would you mind if a posted a version of it on the bricolagecms.org site, or would you mind sending me a version in "blog post" format?

Phillip.

On 2010-10-12, at 10:07 PM, Matthew Rolf wrote:

> Over the last few months I migrated a site I do on the side from Blogger to Bricolage 2.0. In addition to the migration, I did a full site redesign, which you can read about below if things like that interest you. The topic of the site is American soccer.
>
> http://soccer.fakesigi.com/
> http://soccer.fakesigi.com/about_fake_sigi_website_redesign.html
>
> I also built the templates from scratch, and when I get a few moments, I'm going to contribute them back to the community with instructions on how to use them. I think they would be very good for a "beginner" site, in that they can build regular web pages, blog posts, keyword archives, a sitemap, and an Atom feed. If you look at the source code, you'll see the HTML is very clean and concise. The templates are the same, and should be understandable for newbies.
>
> First, what's it like running Bric 2? Much nicer than Bric 1 in many ways. The install is much easier. The new desk interface and content editing interface are very much improved over 1.10. The new element constraints are a great feature. Apache 2 and Modperl 2 usage. I have not tried Mysql, so I can not report on that. But from an editor's perspective, this is a much friendlier release. Also, the new awareness in the burner to not republish repeated pages is fantastic.
>
> As for the bad - I am currently using JS-Quicktags to get around issues with the Xinha editor. Some of you may not find that disappointing, others will retch. Document publication is still a bottle neck, as I can burn around 800 pages in 5 minutes, only to wait 20-25 minutes for those to publish out over SFTP. I had to apply some patches that are not in 2.0 yet to get SFTP publishing to work, with the gracious advice of my former colleagues at Denison, and David actually has rewritten the framework from the ground up for the next release. Also, I have been unable to get any other browser other than Firefox to work, as I've documented on the list. I've found a few other minor bugs which will hopefully get patched before the next release.
>
> The migration itself was not bad. I got the XML dump of my blog from Blogger, and used a perl script with XML::XSLT and an XSL Template to convert the Blogger XML into something SOAP-friendly. However, I also had to run some regex to clean up some of the code. In particular, google has taken to dumping a unique ID into the content field of their blogs, which was a little tricky to get out.
>
> Once I had the SOAP friendly XML, I imported the content without much issue. I then set about tweaking the design, implementing a KinoSearch search engine, and doing a whole bunch of keyword edits. I worked on the transition from May through August, and Bricolage work (install, tuning, document modeling, templating, permissions) took probably only a quarter of the time. My goal was to keep it very simple.
>
> Probably the most interesting thing I did template wise was with the keywords. For any keyword added to a story, on publish I look for a keyword archive story with a corresponding title and publish that as well. Keyword stories must be added manually, so keywords can not be added on the fly when editing a story. I knew Phillip has some template code to automate this, but as I mentioned to David, I think Keyword archive-story management code built-in to the system would be really useful.
>
> For hardware I'm running two machines. The Postgresql database is on a 1.6Ghz Core 2 MacMini with a 250GB HD and 2GB RAM. Bricolage is on a 2.0Ghz single-core Athlon from early 2005, with a 80GB HD and 1GB RAM. This setup works very well, as giving Postgres it's own machine does wonders in freeing up Bricolage from a lot of I/O issues. The MacMini never registers over 30% processor load in top, and is probably HD-bound. The Bricolage machine could benefit from another HD or disk array for burning and moving pages, and a single core would not be enough for a serious shop, as apache and bric_queued start competing for resources. Again, SFTP publishing is far and away the biggest performance bottleneck.
>
> The migration was about 800 stories - I did not do any media. It was very simple, and I kept the feature list down so as to not complicate it. Once the new site was up and beta tested for about three weeks, migrating was as simple as redirecting the DNS to the new site. There were no issues upon startup, and I've just had to tweak a few CSS rules.
>
> Overall, I'm very pleased with Bric 2. Let me know if you have questions or if I left anything out here. I'm up for Bric 2 migration consulting if anyone has a gig. ;)
>
> -Matt

--
Phillip Smith // Simplifier of Technology // COMMUNITY BANDWIDTH
www.communitybandwidth.ca // www.phillipadsmith.com

david at kineticode

Oct 18, 2010, 12:02 PM

Post #3 of 6 (716 views)
Permalink

Re: Blogger to Bricolage 2.0 migration [In reply to]

Great post Matt, thanks. I agree with Phillip that it would be interesting to see as a blog post.

Some comments below.

On Oct 12, 2010, at 7:07 PM, Matthew Rolf wrote:

> Over the last few months I migrated a site I do on the side from Blogger to Bricolage 2.0. In addition to the migration, I did a full site redesign, which you can read about below if things like that interest you. The topic of the site is American soccer.
>
> http://soccer.fakesigi.com/
> http://soccer.fakesigi.com/about_fake_sigi_website_redesign.html
>
> I also built the templates from scratch, and when I get a few moments, I'm going to contribute them back to the community with instructions on how to use them. I think they would be very good for a "beginner" site, in that they can build regular web pages, blog posts, keyword archives, a sitemap, and an Atom feed. If you look at the source code, you'll see the HTML is very clean and concise. The templates are the same, and should be understandable for newbies.

That would rock.

> First, what's it like running Bric 2? Much nicer than Bric 1 in many ways. The install is much easier. The new desk interface and content editing interface are very much improved over 1.10. The new element constraints are a great feature. Apache 2 and Modperl 2 usage. I have not tried Mysql, so I can not report on that. But from an editor's perspective, this is a much friendlier release. Also, the new awareness in the burner to not republish repeated pages is fantastic.

Yay!

> As for the bad - I am currently using JS-Quicktags to get around issues with the Xinha editor. Some of you may not find that disappointing, others will retch.

Is that this bug?

http://bricolage.lighthouseapp.com/projects/29601/tickets/180

Or is there something else we should know about?

> Document publication is still a bottle neck, as I can burn around 800 pages in 5 minutes, only to wait 20-25 minutes for those to publish out over SFTP. I had to apply some patches that are not in 2.0 yet to get SFTP publishing to work, with the gracious advice of my former colleagues at Denison, and David actually has rewritten the framework from the ground up for the next release.

Net::SSH2 needs some significant reworking, frankly. It's quite broken. See https://rt.cpan.org/Ticket/Display.html?id=59332. The underlying libssh2 library is really nice, so there's plenty of room here to get things fixed.

OTOH, if this is a personal project, why use SFTP to distribute stuff?

> Also, I have been unable to get any other browser other than Firefox to work, as I've documented on the list. I've found a few other minor bugs which will hopefully get patched before the next release.

Is there a bug report for that, too?

> The migration itself was not bad. I got the XML dump of my blog from Blogger, and used a perl script with XML::XSLT and an XSL Template to convert the Blogger XML into something SOAP-friendly. However, I also had to run some regex to clean up some of the code. In particular, google has taken to dumping a unique ID into the content field of their blogs, which was a little tricky to get out.

Buggers.

> Once I had the SOAP friendly XML, I imported the content without much issue. I then set about tweaking the design, implementing a KinoSearch search engine, and doing a whole bunch of keyword edits. I worked on the transition from May through August, and Bricolage work (install, tuning, document modeling, templating, permissions) took probably only a quarter of the time. My goal was to keep it very simple.

Impressive that the Bricolage work took such a small percentage of the time. Well done.

> Probably the most interesting thing I did template wise was with the keywords. For any keyword added to a story, on publish I look for a keyword archive story with a corresponding title and publish that as well.

Huh. When I've done this before, I used a story type + slug, as a keyword usually makes an ideal slug.

> Keyword stories must be added manually, so keywords can not be added on the fly when editing a story. I knew Phillip has some template code to automate this, but as I mentioned to David, I think Keyword archive-story management code built-in to the system would be really useful.

Ideally such a thing would be generalized, because folks might want similar functionality for contributors or categories. See http://justatheory.com/bricolage/design/tasks_jobs_actions.html for a design.

> For hardware I'm running two machines. The Postgresql database is on a 1.6Ghz Core 2 MacMini with a 250GB HD and 2GB RAM. Bricolage is on a 2.0Ghz single-core Athlon from early 2005, with a 80GB HD and 1GB RAM. This setup works very well, as giving Postgres it's own machine does wonders in freeing up Bricolage from a lot of I/O issues. The MacMini never registers over 30% processor load in top, and is probably HD-bound.

I/O is usually the issue for a database. Although, frankly, for a site the size of yours, I would think that the entire database could fit into memory. How big is your database? And did you tune settings in postgresql.conf?

> The Bricolage machine could benefit from another HD or disk array for burning and moving pages, and a single core would not be enough for a serious shop, as apache and bric_queued start competing for resources. Again, SFTP publishing is far and away the biggest performance bottleneck.
>
> The migration was about 800 stories - I did not do any media. It was very simple, and I kept the feature list down so as to not complicate it. Once the new site was up and beta tested for about three weeks, migrating was as simple as redirecting the DNS to the new site. There were no issues upon startup, and I've just had to tweak a few CSS rules.

Nice.

> Overall, I'm very pleased with Bric 2. Let me know if you have questions or if I left anything out here. I'm up for Bric 2 migration consulting if anyone has a gig. ;)

Good luck!

Best,

David

phillip at communitybandwidth

Oct 28, 2010, 5:22 AM

Post #4 of 6 (681 views)
Permalink

Re: Blogger to Bricolage 2.0 migration [In reply to]

I'm going to summarize on Matt's behalf, if I don't get a blog post from him soon! ;)

On 2010-10-18, at 3:02 PM, David E. Wheeler wrote:

> Great post Matt, thanks. I agree with Phillip that it would be interesting to see as a blog post.
>
> Some comments below.
>
> On Oct 12, 2010, at 7:07 PM, Matthew Rolf wrote:
>
>> Over the last few months I migrated a site I do on the side from Blogger to Bricolage 2.0. In addition to the migration, I did a full site redesign, which you can read about below if things like that interest you. The topic of the site is American soccer.
>>
>> http://soccer.fakesigi.com/
>> http://soccer.fakesigi.com/about_fake_sigi_website_redesign.html
>>
>> I also built the templates from scratch, and when I get a few moments, I'm going to contribute them back to the community with instructions on how to use them. I think they would be very good for a "beginner" site, in that they can build regular web pages, blog posts, keyword archives, a sitemap, and an Atom feed. If you look at the source code, you'll see the HTML is very clean and concise. The templates are the same, and should be understandable for newbies.
>
> That would rock.
>
>> First, what's it like running Bric 2? Much nicer than Bric 1 in many ways. The install is much easier. The new desk interface and content editing interface are very much improved over 1.10. The new element constraints are a great feature. Apache 2 and Modperl 2 usage. I have not tried Mysql, so I can not report on that. But from an editor's perspective, this is a much friendlier release. Also, the new awareness in the burner to not republish repeated pages is fantastic.
>
> Yay!
>
>> As for the bad - I am currently using JS-Quicktags to get around issues with the Xinha editor. Some of you may not find that disappointing, others will retch.
>
> Is that this bug?
>
> http://bricolage.lighthouseapp.com/projects/29601/tickets/180
>
> Or is there something else we should know about?
>
>> Document publication is still a bottle neck, as I can burn around 800 pages in 5 minutes, only to wait 20-25 minutes for those to publish out over SFTP. I had to apply some patches that are not in 2.0 yet to get SFTP publishing to work, with the gracious advice of my former colleagues at Denison, and David actually has rewritten the framework from the ground up for the next release.
>
> Net::SSH2 needs some significant reworking, frankly. It's quite broken. See https://rt.cpan.org/Ticket/Display.html?id=59332. The underlying libssh2 library is really nice, so there's plenty of room here to get things fixed.
>
> OTOH, if this is a personal project, why use SFTP to distribute stuff?
>
>> Also, I have been unable to get any other browser other than Firefox to work, as I've documented on the list. I've found a few other minor bugs which will hopefully get patched before the next release.
>
> Is there a bug report for that, too?
>
>> The migration itself was not bad. I got the XML dump of my blog from Blogger, and used a perl script with XML::XSLT and an XSL Template to convert the Blogger XML into something SOAP-friendly. However, I also had to run some regex to clean up some of the code. In particular, google has taken to dumping a unique ID into the content field of their blogs, which was a little tricky to get out.
>
> Buggers.
>
>> Once I had the SOAP friendly XML, I imported the content without much issue. I then set about tweaking the design, implementing a KinoSearch search engine, and doing a whole bunch of keyword edits. I worked on the transition from May through August, and Bricolage work (install, tuning, document modeling, templating, permissions) took probably only a quarter of the time. My goal was to keep it very simple.
>
> Impressive that the Bricolage work took such a small percentage of the time. Well done.
>
>> Probably the most interesting thing I did template wise was with the keywords. For any keyword added to a story, on publish I look for a keyword archive story with a corresponding title and publish that as well.
>
> Huh. When I've done this before, I used a story type + slug, as a keyword usually makes an ideal slug.
>
>> Keyword stories must be added manually, so keywords can not be added on the fly when editing a story. I knew Phillip has some template code to automate this, but as I mentioned to David, I think Keyword archive-story management code built-in to the system would be really useful.
>
> Ideally such a thing would be generalized, because folks might want similar functionality for contributors or categories. See http://justatheory.com/bricolage/design/tasks_jobs_actions.html for a design.
>
>> For hardware I'm running two machines. The Postgresql database is on a 1.6Ghz Core 2 MacMini with a 250GB HD and 2GB RAM. Bricolage is on a 2.0Ghz single-core Athlon from early 2005, with a 80GB HD and 1GB RAM. This setup works very well, as giving Postgres it's own machine does wonders in freeing up Bricolage from a lot of I/O issues. The MacMini never registers over 30% processor load in top, and is probably HD-bound.
>
> I/O is usually the issue for a database. Although, frankly, for a site the size of yours, I would think that the entire database could fit into memory. How big is your database? And did you tune settings in postgresql.conf?
>
>> The Bricolage machine could benefit from another HD or disk array for burning and moving pages, and a single core would not be enough for a serious shop, as apache and bric_queued start competing for resources. Again, SFTP publishing is far and away the biggest performance bottleneck.
>>
>> The migration was about 800 stories - I did not do any media. It was very simple, and I kept the feature list down so as to not complicate it. Once the new site was up and beta tested for about three weeks, migrating was as simple as redirecting the DNS to the new site. There were no issues upon startup, and I've just had to tweak a few CSS rules.
>
> Nice.
>
>> Overall, I'm very pleased with Bric 2. Let me know if you have questions or if I left anything out here. I'm up for Bric 2 migration consulting if anyone has a gig. ;)
>
> Good luck!
>
> Best,
>
> David
>

--
Phillip Smith // Simplifier of Technology // COMMUNITY BANDWIDTH
www.communitybandwidth.ca // www.phillipadsmith.com

mattrolf at me

Oct 28, 2010, 11:37 AM

Post #5 of 6 (677 views)
Permalink

Re: Blogger to Bricolage 2.0 migration [In reply to]

On Oct 18, 2010, at 3:02 PM, David E. Wheeler wrote:

>> As for the bad - I am currently using JS-Quicktags to get around issues with the Xinha editor. Some of you may not find that disappointing, others will retch.
>
> Is that this bug?
>
> http://bricolage.lighthouseapp.com/projects/29601/tickets/180
>
> Or is there something else we should know about?

No, it's that bug.

> Net::SSH2 needs some significant reworking, frankly. It's quite broken. See https://rt.cpan.org/Ticket/Display.html?id=59332. The underlying libssh2 library is really nice, so there's plenty of room here to get things fixed.

Yeah, I saw that. Fun.

>
> OTOH, if this is a personal project, why use SFTP to distribute stuff?

The public web server is remote from the Bricolage location. So I want to make sure it is secure. Only SSH will pick up calls, and you need a key.

>> Also, I have been unable to get any other browser other than Firefox to work, as I've documented on the list. I've found a few other minor bugs which will hopefully get patched before the next release.
>
> Is there a bug report for that, too?

I have not filed a bug yet. I've brought it up on this list, but I'm not sure how to duplicate it yet. Although it's persistent enough that I think it would be good to file a bug.

> Impressive that the Bricolage work took such a small percentage of the time. Well done.

Obviously, I'm familiar with the product. That helps. But

>> Probably the most interesting thing I did template wise was with the keywords. For any keyword added to a story, on publish I look for a keyword archive story with a corresponding title and publish that as well.
>
> Huh. When I've done this before, I used a story type + slug, as a keyword usually makes an ideal slug.

You could do that, too. Here's my code for a typical article:
<%args>
@klist => $story->get_keywords
...

Then at the bottom of each page:

% foreach (@klist){
<li><a href="http://soccer.fakesigi.com/<% $_->get_sort_name %>/"><% $_->get_screen_name %></a></li>
%}

And then when I'm done:

<%perl>
foreach (@klist) {my $name = ($_->get_screen_name);
push (@keynames,$name);}
foreach (@keynames) {@keystory = Bric::Biz::Asset::Business::Story->list({element_key_name => 'web_keyword_index', title => $_ });

Maybe slug would be more efficient, I don't know.

> Ideally such a thing would be generalized, because folks might want similar functionality for contributors or categories. See http://justatheory.com/bricolage/design/tasks_jobs_actions.html for a design.

Yes, I agree.

> I/O is usually the issue for a database. Although, frankly, for a site the size of yours, I would think that the entire database could fit into memory. How big is your database? And did you tune settings in postgresql.conf?

The current DB is 24MB, so yeah. I take back what I said about the Mini being HD bound. I did tune it as well.

As for having it live with Bricolage, remember that the Bric machine only has 1GB of RAM, and an older processor. It could be done without question but I figured I'd plan for the future at the start.

-Matt

mattrolf at me

Nov 10, 2010, 12:17 PM

Post #6 of 6 (650 views)
Permalink

Re: Blogger to Bricolage 2.0 migration [In reply to]

On Oct 28, 2010, at 2:37 PM, Matthew Rolf wrote:

> Also, I have been unable to get any other browser other than Firefox to work, as I've documented on the list. I've found a few other minor bugs which will hopefully get patched before the next release.
>>
>> Is there a bug report for that, too?
>
> I have not filed a bug yet. I've brought it up on this list, but I'm not sure how to duplicate it yet. Although it's persistent enough that I think it would be good to file a bug.

Right, so all of a sudden, this is no longer a problem. I have no idea what I might have done other than restarting the system.

Maybe a junk router?

Weird.

-Matt

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads