unicode characters in web form aspx pages, utf-8

Discussion in 'ASP .Net' started by ray well, Mar 22, 2005.

  1. ray well

    ray well Guest


    i need to include some non english text in some web from aspx pages, in
    which i used unicode characters and saved them with unicode encoding
    (without signature).

    in regular html only pages i add the following line to the <head> section to
    make the text show up properly, otherwise it shows up as garbage.

    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">

    when i do this in aspx page, the unicode text still shows up as garbage.

    how can i make the unicode show properly?

    please include a snipett of sample.


    ray well, Mar 22, 2005
    1. Advertisements

  2. ray well

    Joerg Jooss Guest

    META tags are just a last resort, and they are plain useless if the
    page's encoding isn't the supposed encoding.

    Are you really using UTF-8 as response encoding and is your browser
    configured to pick up the encoding automatically?

    Joerg Jooss, Mar 22, 2005
    1. Advertisements

  3. ray well

    ray well Guest

    i'm not using the unicode in response code, but in labels indentifying areas
    on the page (which shows up as garbage. my browser can read the encoding in
    regular html pages, so it is configured to read the encoding.

    ray well, Mar 22, 2005
  4. If you are using characters in the range 128-255,
    you need to use iso-8859-1 as your encoding, and not utf-8.
    Juan T. Llibre, Mar 22, 2005
  5. ray well

    ray well Guest

    i'm not using char 128-255, i need to use unicode char.

    ray well, Mar 23, 2005
  6. Have you set the encoding settings in web.config ?

    fileEncoding="utf-8" />
    Juan T. Llibre, Mar 23, 2005
  7. ray well

    ray well Guest


    thanks for

    <globalization requestEncoding="utf-8" responseEncoding="utf-8"
    fileEncoding="utf-8" />

    i was missing fileEncoding="utf-8", i had the other 2, when i added this it
    started working.

    thanks again

    ray well, Mar 23, 2005
  8. "It's the little things that getcha."
    -unknown philosopher

    Glad it's working now!
    Juan T. Llibre, Mar 23, 2005
  9. ray well


    May 14, 2009
    Likes Received:
    I’ve been working on a procedure in order to insert data from a Web Service to a SQL Server using a Bulk Insert.

    Something like this:

    Line1 DataSet dsAd_Mstr = objWebServices.Obtener_Ad_Mstr();
    Line2 objConnectionC.Open();
    Line3 Global.BulkTableInsert(dsAd_Mstr, objConnectionC, "Ad_Mstr");

    In the first line I’m extracting the information from a Web Service. This will return something like this (take a look at the encoding, is it ok ?):

    <?xml version="1.0" encoding="utf-8"?>
    <DataSet xmlns="tempuri.org/">
    <xs:schema id="NewDataSet" xmlns="" xmlns:xs="w3/2001/XMLSchema" xmlns:msdata="urn:schemas-microsoft-com:xml-msdata">
    <xs:element name="NewDataSet" msdata:IsDataSet="true">
    <xs:complexType> <xs:choice maxOccurs="unbounded">
    <xs:element name="ad_mstr"> <xs:complexType> <xs:sequence>
    <xs:element name="ad_addr" type="xs:string" minOccurs="0" />
    <xs:element name="ad_name" type="xs:string" minOccurs="0" />
    <xs:element name="ad_line1" type="xs:string" minOccurs="0" />
    <xs:element name="ad_domain" type="xs:string" MinOccurs="0" />
    <xs:element name="oid_ad_mstr" type="xs:decimal" minOccurs="0" />
    </xs:sequence> </xs:complexType> </xs:element>
    </xs:choice> </xs:complexType> </xs:element> </xs:schema>
    <diffgr:diffgram xmlns:msdata="urn:schemas-microsoft-com:xml-msdata" xmlns:diffgr="urn:schemas-microsoft-com:xml-diffgram-v1">
    <NewDataSet xmlns="">
    <ad_mstr diffgr:id="ad_mstr4073" msdata:rowOrder="4072">
    <ad_name>COMPANY X</ad_name>
    <ad_line1 />
    <ad_line2 />
    <ad_city />
    <ad_state />
    <ad_zip />
    <ad_mstr diffgr:id="ad_mstr4074" msdata:rowOrder="4073">
    <ad_name>COMPANY Y</ad_name>
    <ad_line1 />
    I think there is no need to explain line number 2 (opening connection).

    The line number 3 is a calling to the following function:
    Line21 public static void BulkTableInsert(DataSet objDS, SqlConnection objCon, string tablename)
    Line22 {
    Line23 //Change the column mapping first.
    Line24 System.Text.StringBuilder sb = new System.Text.StringBuilder( 1000);
    Line25 System.IO.StringWriter sw = new System.IO.StringWriter(sb);
    Line26 foreach( DataColumn col in objDS.Tables[tablename].Columns)
    Line27 {
    Line28 col.ColumnMapping = System.Data.MappingType.Attribute;
    Line29 }
    Line31 objDS.WriteXml(sw, System.Data.XmlWriteMode.WriteSchema);
    Line32 string sqlText = buildBulkUpdateSql(sb.ToString(), objDS.Tables[tablename]);
    Line34 sqlText = sqlText.Replace("''true''", "''1''");
    Line35 sqlText = sqlText.Replace("''false''", "''0''");
    Line36 sqlText = sqlText.Replace("á", "a");
    Line37 sqlText = sqlText.Replace("é", "e");
    Line38 sqlText = sqlText.Replace("í", "i");
    Line39 sqlText = sqlText.Replace("ó", "o");
    Line40 sqlText = sqlText.Replace("ú", "u");
    Line41 sqlText = sqlText.Replace("Ñ", "N");
    Line42 sqlText = sqlText.Replace("ñ", "n");
    Line43 execSql(objCon, sqlText);
    Line44 }
    (I implemented by myself lines 34 to 42, are they ok ? If I configure in a different way the web service could I avoid this ?)

    The line 32 is calling the following function:
    Line45 static string buildBulkUpdateSql( string dataXml, DataTable table)
    Line46 {
    Line47 System.Text.StringBuilder sb = new System.Text.StringBuilder();
    Line48 dataXml = dataXml.Replace(Environment.NewLine, "");
    Line49 dataXml = dataXml.Replace("\"", "''");
    Line50 //init the xml doc
    Line51 sb.Append(" SET NOCOUNT ON");
    Line52 sb.Append(" DECLARE @hDoc INT");
    Line53 //sb.AppendFormat(" EXEC sp_xml_preparedocument @hDoc OUTPUT, '<?xml version=''1.0'' encoding=''iso-8859-1''?> {0}'", dataXml);
    Line54 sb.AppendFormat(" EXEC sp_xml_preparedocument @hDoc OUTPUT, '{0}'", dataXml);
    Line55 //This code deletes old data based on PK.
    Line56 sb.AppendFormat(" DELETE {0} FROM {0} INNER JOIN ", table.TableName);
    Line57 sb.AppendFormat(" (SELECT * FROM OPENXML (@hdoc, '/NewDataSet/{0}', 1)",
    Line58 table.TableName);
    Line59 sb.AppendFormat(" WITH {0}) xmltable ON 1 = 1", table.TableName);
    Line60 foreach( DataColumn col in table.PrimaryKey)
    Line61 {
    Line62 sb.AppendFormat(" AND {0}.{1} = xmltable.{1}", table.TableName,
    Line63 col.ColumnName);
    Line64 }
    Line65 //This code inserts new data.
    Line66 sb.AppendFormat(" INSERT INTO {0} SELECT *", table.TableName);
    Line67 sb.AppendFormat(" FROM OPENXML (@hdoc, '/NewDataSet/{0}', 1) WITH {0}",
    Line68 table.TableName);
    Line69 //clear the xml doc
    Line70 sb.Append(" EXEC sp_xml_removedocument @hDoc");
    Line71 return sb.ToString();
    Line72 }

    I think that you can imagine what does the function called in line 43 (Executes the dynamic query).

    I’m taking these 3 last function from the article "A generic bulk insert using DataSets and OpenXML" from CodeProject.

    So far the whole procedure looks great. But when I execute the calling to the line 3, an exception throws stating just an SQL Exception.

    The code generated (sqlText) by the BulkTableInsert looks like this (nevermind about the linebreaks, they're there because of the cut-copy operation, they are not for real):

    SET NOCOUNT ON DECLARE @hDoc INT EXEC sp_xml_preparedocument @hDoc OUTPUT,
    '<NewDataSet> <xs:schema id=''NewDataSet'' xmlns='''' xmlns:xs=''w3/2001/XMLSchema'' xmlns:msdata=''urn:schemas-microsoft-com:xml-msdata''> <xs:element name=''NewDataSet'' msdata:IsDataSet=''1''> <xs:complexType> <xs:choice maxOccurs=''unbounded''> <xs:element name=''ad_mstr''> <xs:complexType> <xs:attribute name=''ad_addr'' type=''xs:string'' /> <xs:attribute name=''ad_name'' type=''xs:string'' /> <xs:attribute name=''ad_line1'' type=''xs:string'' />
    <xs:attribute name=''oid_ad_mstr'' type=''xs:decimal'' /> </xs:complexType> </xs:element> </xs:choice> </xs:complexType> </xs:element> </xs:schema>
    <ad_mstr ad_addr=''SOCIOS3'' ad_name=''COMPANY X'' ad_line1='''' ad_line2='''' ad_city='''' ad_state='''' ad_z
    ip='''' ad_type=''customer'' ad_attn='''' ad_phone='''' ad_ext='''' ad_ref='''' ad_sort=''COMPANY X'' ad_country=''MEXICO'' ad_attn2='''' ad_phone2='''' ad_ext2='''' ad_fax='''' ad_fax2='''' ad_line3='''' ad_user1='''' ad_user2='''' ad_lang='''' ad_pst_id='''' ad_date=''02/03/1993'' ad_county='''' ad_temp=''0'' ad_bk_acct1='''' ad_bk_acct2='''' ad_format=''0'' ad_vat_reg='''' ad_coc_reg='''' ad_gst_id='''' ad_tax_type='''' ad_taxc=''1'' ad_taxable=''1'' ad_tax_in=''0'' ad_conrep='''' ad_edi_tpid='''' ad_edi_ctrl='';;;;'' ad_timezone='''' ad_userid='''' ad_edi_id='''' ad_barlbl_prt='''' ad_barlbl_val='''' ad_calendar='''' ad_edi_std='''' ad_edi_level='''' ad__qad01='''' ad__qad02='''' ad__qad03='''' ad__qad04='''' ad__qad05='''' ad__chr01='''' ad__chr02='''' ad__chr03='''' ad__chr04='''' ad__chr05='''' ad_tp_loc_code='''' ad_ctry=''MEX'' ad_tax_zone=''MEXICO'' ad_tax_usage=''BUSES'' ad_misc1_id='''' ad_misc2_id='''' ad_misc3_id='''' ad_wk_offset=''0'' ad_inv_mthd='''' ad_sch_mthd='''' ad_po_mt
    hd='''' ad_asn_data='''' ad_intr_division='''' ad_tax_report=''0'' ad_name_control='''' ad_last_file=''0'' ad_domain=''VIM'' oid_ad_mstr=''0'' />
    DELETE ad_mstr FROM ad_mstr INNER JOIN (SELECT * FROM OPENXML (@hdoc, '/NewDataSet/ad_mstr', 1) WITH ad_mstr) xmltable ON 1 = 1 INSERT INTO ad_mstr SELECT * FROM OPENXML (@hdoc, '/NewDataSet/ad_mstr', 1) WITH ad_mstr EXEC sp_xml_removedocument @hDoc

    If I take the sqlText (line 43) value (the whole XML code whose fragment I pasted above) and test it in an SQL Server Query Analyzer, first I have change all the apostrophes inside the text values (change ARTHUR’S for ARTHUR&apos;S, is this correct ?), eliminated the linebreaks, then I have an error stating:

    Server: Msg 6603, Level 16, State 1, Procedure sp_xml_preparedocument, Line 1 XML parsing error: An invalid character was found in text content.

    Now I have some "special" (and hidden) characters. I can remove the whole text in the value field and everything works fine, so that's what I'm guessing there are some hidden chars.

    How can I do to remove them ?
    Should I prepare the info prior to send it to the web service ?
    Should I configure the web service in order to fix this ?
    Should I chance the encoding ot he collation ?

    The server (and the table) where I’m extracting the info is the same that the server (and the table) where I’m trying to insert the info.
    I have changed the links because of this site requirements.
    If you need extra info, please, let me know.
    Thanks a lot.
    xavirm, May 14, 2009
  10. ray well


    Jun 1, 2009
    Likes Received:

    I am using asp.net 2.0.In this I need get all the symbols to be displayed.
    Currently I am getting that by using the unicode in code behind. But in that some symbols like "Miscellaneous Technical" are not coming.

    <globalization requestEncoding="utf-8" responseEncoding="utf-8" /> this is my blobalization in web.config.

    I am very new to this please help me as soon as possible. Its very urgent..

    Thank you..
    Swan, Jun 1, 2009
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.