Unicode Support In SAP WAS.E

Unicode Support in SAPWeb Application ServerDr. Christian HansenMatthias MittelsteinServer Technology Internationalization SAP AG

AgendaScripts, Characters and Code Pages Conventional code pages UnicodeUnicode in the Web Application Server Front End Communication Application server Database PrintingConversion to Unicode C and C programs ABAP programs Database System landscapeAvailability and Release PlanningSummary 2002 SAP Labs, LLC, WAS203, Christian Hansen 2

Scripts, Characters and Code Pages: FinnishFrench, ItalianNorwegianPortugueseSpanishSwedishTurkish 2002 SAP Labs, LLC, WAS203, Christian Hansen Slovene

Conventional Code Pages: FinnishFrench, ItalianNorwegianPortugueseSpanishSwedishTurkish 2002 SAP Labs, LLC, WAS203, Christian Hansen Slovene

Conventional Code Pages FinnishFrench, ItalianNorwegianPortugueseSpanishSwedishTurkish 2002 SAP Labs, LLC, WAS203, Christian Hansen Slovene

Conventional Code Pages: FinnishFrench, ItalianNorwegianPortugueseSpanishSwedishTurkish 2002 SAP Labs, LLC, WAS203, Christian Hansen Slovene

Conventional Code Pages: FinnishFrench, ItalianNorwegianPortugueseSpanishSwedishTurkish 2002 SAP Labs, LLC, WAS203, Christian Hansen Slovene

Conventional Code Pages: FinnishFrench, ItalianNorwegianPortugueseSpanishSwedishTurkish 2002 SAP Labs, LLC, WAS203, Christian Hansen Slovene

Conventional Code Pages: nishFrench, ItalianNorwegianPortugueseSpanishSwedishTurkish 2002 SAP Labs, LLC, WAS203, Christian Hansen Slovene

Conventional Code Pages: manFinnishFrench, ItalianNorwegianPortugueseSpanishSwedishTurkish 2002 SAP Labs, LLC, WAS203, Christian Hansen nSlovene

Conventional Code Pages: Earlier SAP approachesEarlier SAP approaches used a fixed mappingfrom database to code page "single code page system“from application server to code page "MNLS“ (obsolete) R/3 releases 2.2F to 3.0Cfrom language to code page "MDMP“ (current SAP technique for using multiple codepages) Using tables TCP0C, TCP0D, TCPDB 2002 SAP Labs, LLC, WAS203, Christian Hansen 11

Conventional Code Pages: Earlier SAP approaches(MDMP)West European View 2002 SAP Labs, LLC, WAS203, Christian Hansen 12Japanese ViewKorean View

Conventional Code Pages: Earlier SAP approachesEarlier SAP approaches had disadvantages when the numberof concurrent system code pages increases: ambiguous data, when accessing across a code pageboundary data encoding hard to understand for non-SAP programs sophisticated programming techniques needed to handledata in the appropriate code page each user limited to her own language 2002 SAP Labs, LLC, WAS203, Christian Hansen 13

Solution: nnishFrench, ItalianNorwegianPortugueseSpanishSwedishTurkish 2002 SAP Labs, LLC, WAS203, Christian Hansen nSloveneAnd morelanguagescan besupportedeasilywithout theneed fornew codepages orother newmethods

Solution: Unicode charactersASCIIGeneral ScriptsSymbolsCJK Ideographs65,000 charactersHangulCompatibilitySurrogate Area 2002 SAP Labs, LLC, WAS203, Christian Hansen 15Additional1,000,000 characters

Representation of Unicode CharactersUTF-16 – Unicode Transformation Format, 16 bit encoding Fixed length, 1 character 2 bytes (surrogate pairs 2 2 bytes) Platform-dependent byte order (big/little endian) 2 byte alignment restrictionUTF-8 – Unicode Transformation Format, 8 bit encoding Variable length, 1 character 1.4 bytes Platform independent no alignment restriction 7 bit US ASCII compatibleCharacterUnicodescalar valueUTF-16big endianUTF-16little endianaU 006100 6161 0061äU 00E400 E4E4 00C3 A4αU 03B103 B1B1 03CE B1U 347934 7979 34E3 91 B9 2002 SAP Labs, LLC, WAS203, Christian Hansen 16UTF-8

AgendaScripts, Characters and Code Pages Conventional code pages UnicodeUnicode in the Web Application Server Front End Communication Application server Database PrintingConversion to Unicode C and C programs ABAP programs Database System landscapeAvailability and Release PlaningSummary 2002 SAP Labs, LLC, WAS203, Christian Hansen 17

Unicode in the SAP Web Application Server SAP GUIUnicode Release 6.20 GUIs use UTF-8 forcommunication and UTF-8 and UTF-16internally RFC and other: RFC, XML and other:Implicit reinterpretations andCode page conversions onmemcopy based loopholes createcharacter data are explicit andambiguous datamandatory ASCII, DBCS, EBCDIC, MDMP,blended code pages, UTF-16 ASCII, DBCS, EBCDIC, MDMP, . Unicode:PrintingDatabaseCommunication WinGUI 6.30 will use UTF-16 internallyAppl.ServerFront-endnon-Unicode Or only bytes? One standard printer type foreach code page UTF-8 printer for all code pages 2002 SAP Labs, LLC, WAS203, Christian Hansen 18UTF-16 CESU-8 chosen byDBMS vendor UTF-8 printer to cover all characters Normal printers restricted to localtexts with reduced character set

Unicode in the Front-end: SAPGUIUse a 6.20 SAPGUI when working on a Unicode system.a) Frontend code page is set to 4110 (UTF-8)b) Multibyte functionality is activated (see note 508854)a)b)The feature to use Unicode is built into the SAPGUIs. Noseparate executable or separate installation is necessary.6.20-SAPGUIs use UTF-8 for communication when inUnicode mode. 2002 SAP Labs, LLC, WAS203, Christian Hansen 19

SAPGUI for the Windows EnvironmentWinGUI 6.20 delivered Screenshot:Version 6.20 Revision 2Patch level 18 Standard installation 2002 SAP Labs, LLC, WAS203, Christian Hansen 20

SAPGUI for the Java EnvironmentJavaGUI 6.20 delivered Screenshot:Version 6.20 Revision 4 Standard installation Plus modifiedC:\program files\JavaSoft\JRE\1.3.1 02\lib\font.properties 2002 SAP Labs, LLC, WAS203, Christian Hansen 21

SAPGUI for the HTML EnvironmentWebGUI prototype Version 6.20 Remember:WAS 6.20 comes with ITS 6.10 2002 SAP Labs, LLC, WAS203, Christian Hansen 22

Unicode and RFC(1)RFC non-Unicode – non-Unicode Long running standard Receiver converts code pages, if it canRFC Unicode – Unicode No problemRFC Unicode -- non-Unicode The Unicode-side converts from/to the old code page In Unicode - MDMP communication data is interpreted usinglanguage key information System settings allow to catch or ignore possible conversionproblems Applications have to readjust fields, if structured data has beentransported stored in character containersWhen the data has a complicated structure RFC already usesXML und UTF-8 since release 4.6C 2002 SAP Labs, LLC, WAS203, Christian Hansen 23

Unicode and RFC A Unicode receiver canreceive all characters.( solid lines ) A non-Unicode receivercannot receive charactersthat are not in its own codepage. But as long as yourestrict the character set,data can be sent fromeverywhere to everywhere.( dotted lines ) 2002 SAP Labs, LLC, WAS203, Christian Hansen 24(2)

Unicode in the application serversApplication servers use UTF-16 not UCS-4 or UTF-32, because that would be too expensive(memory, bandwidth) not UTF-8 because the dynamic length and offset would be toocomplicated (CPU consumption, robustness of applications) ABAP like JAVAKernel has only one C / C sourceApplications have only one ABAP sourceAdditional hardware requirement: CPU30-35% Memory50% 2002 SAP Labs, LLC, WAS203, Christian Hansen 25

Unicode in the databasesDatabases use UTF-16 or CESU-8* internally Hidden by database client software library The library interface to the kernel uses UTF-16.Additional hardware requirement: UTF-8/CESU-8 UTF-1636%60-70%OracleCESU-8MS SQL ServerUTF-16DB/2 6000UTF-8DB/2 400UTF-16DB/2 390later: UTF-8SAPDB 7.0UTF-16Informix- not supported -See note 379940 for current status. 2002 SAP Labs, LLC, WAS203, Christian Hansen 26* CESU-8 is similar toUTF-8, but when binarysorting is used, it givesthe same result asbinary sorting on UTF16BE. The differencebetween UTF-8 andCESU-8 is visible onlyfor surrogate pairs.

Unicode and printers: LEXMARK UTF-8 printerPrinter of choice when connected to a Unicode systemCan handle any single language in an MDMP systemCan print mixed languages in an MDMP system, whenSAPscript is used carefully. (See scan of printout.) 2002 SAP Labs, LLC, WAS203, Christian Hansen 27

AgendaScripts, Characters and Code Pages Conventional code pages UnicodeUnicode in the Web Application Server Front End Communication Application server Database PrintingConversion to Unicode C and C programs ABAP programs Database System landscapeAvailability and Release PlaningSummary 2002 SAP Labs, LLC, WAS203, Christian Hansen 28

Conversion of C and C programsSAP : C and C programs, the Kernel (done) Modify sources (with tools ccU and ccQ and some manualchanges)Replace char,strcmp, fopen, freadwithSAP UC,strcmpU, fopenU, freadUorSAP RAW, strcmpR,, freadR Generate backward compatible non-Unicode kernels and UTF-16kernels from the same sources. Suggest char16 t and u"String" to standard organizations. Use ICU in Unicode systems for locale-dependent functionality.The International Components for Unicode (ICU) is a C and C library that provides Unicode supportfunctionality. ICU is a collaborative, open-source development project. It is licensed under the X License. Formore details see the http://oss.software.ibm.com/icu/index.hltml. There is also a parallel ICU for Java : external RFC-applications in C or C Modify sources (with tools ccU and ccQ and some manualchanges, see SAP RFC Software Development Kit documentation forfurther information) 2002 SAP Labs, LLC, WAS203, Christian Hansen 29

Conversion of ABAP programsABAP character data types (C,N,D,T,STRING) areautomatically Unicode in a Unicode system. Major part of ABAP coding is ready for Unicode without anychanges Minor part of ABAP coding has to be adapted to comply withUnicode restrictions ( Workshop ABAP217)Use UCCHECK to activate Unicode syntax check and viewproblematic placesDo runtime tests to detect semantic changes in theapplication. Screen runtime tests with the ABAP CoverageAnalyzer SCOV. 2002 SAP Labs, LLC, WAS203, Christian Hansen 30

Conversion of ABAP programs: UCCHECK 2002 SAP Labs, LLC, WAS203, Christian Hansen 31

Conversion of ABAP programs: SCOV 2002 SAP Labs, LLC, WAS203, Christian Hansen 32

Conversion of the database: Single Code PageSystemBuilding up a Unicode system requires converting allcharacter data in the system to UnicodeIn a single code page system the code page of the characterdata is unambiguous.The conversion is done with R3load: R3load – export (conversion to Unicode is done) R3load – import (Unicode data is imported)Downtime may be reduced withIMIG incremental conversion with minimal downtime (indevelopment) 2002 SAP Labs, LLC, WAS203, Christian Hansen 33

Unicode database conversion and downtimeUnicode systemNon-Unicode systemExport FileDowntimeConversionof restOnline conversion time 2002 SAP Labs, LLC, WAS203, Christian Hansen 34

Conversion of the database: MDMP SystemIn MDMP systems the code page of the character data isambiguous and has to be derived from secondaryinformation: Scan database with R3 transaction SPUMG Findexplicit language fields and hidden language fields Recognizetypical characters, recognize language dependentwords Classifytables and report problematic data Enhance database or give hintsThe conversion is done with R3load: R3load multi-code page export R3load import(plus IMIG) Post conversion repair with SUMG (to correct wrong tableclassifications) 2002 SAP Labs, LLC, WAS203, Christian Hansen 35

Unicode Conversion: System landscapesConvert systems one by one: RFC and other communication between non-Unicode andUnicode systems can do code page conversions wherenecessary Convert data destinations first (only Unicode systems canreceive all data) Think of external files as own systems and determine theircode page. Convert files once or setup conversion duringreading. If you had systems separated by code pages, they can bemigrated into a single system now 2002 SAP Labs, LLC, WAS203, Christian Hansen 36