Gossamer Forum: General: Perl Programming: Just a quickie .. regarding non-english charachers...

Jul 21, 2004, 1:28 AM

Andy

Veteran / Moderator (17297 posts)

Jul 21, 2004, 1:28 AM

Post #1 of 9

Shortcut

Just a quickie .. regarding non-english charachers...

Hi guys. Just a quick one.

Is there a way to decode something like;

World/Espa%c3%b1ol

...to;

World/Español

I'm not quite sure what/how the %c3%b1 is assigned/setup :(

Cheers

Andy (mod)
andy@ultranerds.co.uk

IMPORTANT: I've now moved to ultranerds.co.uk, and the .com will no longer work!
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package (plugins total "value" $3,325 & rising, for just $350)| GLinks ULTRA Package PRO (plugins total "value" $5,625 & rising, for just $500)
Support Forum | Links SQL Plugins | DMOZ Dumps | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Compare our different Plugin packages *new* Free CSS Templates

Jul 21, 2004, 5:32 PM

GClemmons

User / Moderator (314 posts)

Jul 21, 2004, 5:32 PM

Post #2 of 9

Shortcut

Re: [Andy] Just a quickie .. regarding non-english charachers... In reply to

This should do it

Code:
use Unicode::String;  
my $str = 'World/Espa%c3%b1ol';  
$str =~ s/%([a-fA-F0-9]{2})/chr(hex($1))/eg;  
my $ustr = new Unicode::String($str);  
print $ustr->latin1();

If your OS is unicode friendly (like OS X), you can skip the Unicode::String bit and just use the regex.

-gordon

s/(\d{2})/chr($1)/ge + print if $_ = '8284703280698276687967';

Last edited by:

GClemmons: Jul 21, 2004, 5:33 PM

Jul 22, 2004, 1:00 AM

Andy

Veteran / Moderator (17297 posts)

Jul 22, 2004, 1:00 AM

Post #3 of 9

Shortcut

Re: [GClemmons] Just a quickie .. regarding non-english charachers... In reply to

Thanks Gordon. I'll give that a go :)

Cheers

Andy (mod)
andy@ultranerds.co.uk

Jul 22, 2004, 2:21 AM

Andy

Veteran / Moderator (17297 posts)

Jul 22, 2004, 2:21 AM

Post #4 of 9

Shortcut

Re: [GClemmons] Just a quickie .. regarding non-english charachers... In reply to

Thanks. That worked ok, except it looks like the RDF file uses stuff like;

<Topic r:id="Top/World/EspaÃ±ol">

... whats the Ã±? Do you think there is any way to go through the RDF file, and change each non-english charachters correctly? (Unicode::MapUTF kinda works, but if you run it more than once, it seems to get rid of foreign charachters all together ... ie "EspaÃ±ol" would change to "Español", and then once you run it again, it would turn into : Espaol (i.e the missing ñ).

Any ideas?

TIA

Andy (mod)
andy@ultranerds.co.uk

Jul 22, 2004, 2:39 AM

GClemmons

User / Moderator (314 posts)

Jul 22, 2004, 2:39 AM

Post #5 of 9

Shortcut

Re: [Andy] Just a quickie .. regarding non-english charachers... In reply to

Quote:

Yeah, that's what a utf-8 string looks like when viewed via an ascii interface. If you want to see the ascii equiv you can use Unicode::String to convert it. Of course, that is only for latin characters - if you have some non-latin character (like cyrillic or japanese characters) it won't work. If your data is in Unicode format, I would leave it that way unless there is a good reason to switch it. If your presentation layer provides the right headers, everything should come out fine on the browsers.

-gordon

s/(\d{2})/chr($1)/ge + print if $_ = '8284703280698276687967';

Jul 22, 2004, 2:55 AM

Andy

Veteran / Moderator (17297 posts)

Jul 22, 2004, 2:55 AM

Post #6 of 9

Shortcut

Re: [GClemmons] Just a quickie .. regarding non-english charachers... In reply to

Quote:

I would leave it that way unless there is a good reason to switch it. If your presentation layer provides the right headers, everything should come out fine on the browsers.

The problem I am having, is with my DMOZ wizard. I'm working on a cleanup routine, which I'm going to put in it, which looks like this so far;

Code:
sub run_rdf_cleanup { 

  print "Cleaning up RDF file... \n"; 

  `mv content.rdf.u8 content.rdf.u8.2`; 

  open (CONTENT,"content.rdf.u8.2") || die $!; 
  open (WRITEIT,">content.rdf.u8") || die $!; 
    while (<CONTENT>) { 
      if (/[\200-\377]/) {  
         s/([\200-\377]+)/from_utf8({ -string => $1, -charset => 'ISO-8859-1'})/eg;  
      } 
      print WRITEIT $_;        
    } 
  close(WRITEIT); 
  close(CONTENT); 

}

Its far from perfect, but I'm hoping it will do the trick.

As you say though, it won't be able to do BIG5 etc formats.. which is a bit of a pain (fortunatly almost everyone who uses my plugin imports English categories; but there are always a few who would need non-english charachters, such as French, German, Spanish, etc.

Its a tough one.

I guess what I could do is just put something up on the detailed page, letting people know that it WONT be able to import non-UTF8 charachters (sorry, thinking out loud :) ).

Cheers

Andy (mod)
andy@ultranerds.co.uk

Jul 22, 2004, 3:10 AM

Andy

Veteran / Moderator (17297 posts)

Jul 22, 2004, 3:10 AM

Post #7 of 9

Shortcut

Re: [GClemmons] Just a quickie .. regarding non-english charachers... In reply to

Yaaaay... seems to be working :)

I'm gonna give it a few more tests; and if its ok, then I'll get it finished off, and get the update released.

Thanks for your help on this one Gordon :)

Cheers

Andy (mod)
andy@ultranerds.co.uk

Jul 22, 2004, 4:47 AM

GClemmons

User / Moderator (314 posts)

Jul 22, 2004, 4:47 AM

Post #8 of 9

Shortcut

Re: [Andy] Just a quickie .. regarding non-english charachers... In reply to

Any time. Regarding the spanish, french, german, etc characters, you can still convert those to Latin-1 - the problem lies in the non-latin characters like arabic, cyrillic, japanese, etc.

Good luck with the release.

-gordon

s/(\d{2})/chr($1)/ge + print if $_ = '8284703280698276687967';

Jul 22, 2004, 5:03 AM

Andy

Veteran / Moderator (17297 posts)

Jul 22, 2004, 5:03 AM

Post #9 of 9

Shortcut

Re: [GClemmons] Just a quickie .. regarding non-english charachers... In reply to

Quote:

the problem lies in the non-latin characters like arabic, cyrillic, japanese, etc.

Yeah, I'm just going to put something on the site about the arabic, cyrillic, japanese imports.

Thanks again.

Andy (mod)
andy@ultranerds.co.uk