The Always-On Enterprise: Acheiving Higher Service Levels With . - VMware
The Always-On Enterprise: Acheiving Higher Service Levels with VMware Mornay Van Der Walt Vice President, System Architect IXIS Capital Markets
Session Agenda PART 1: The Early Days Company Profile IXIS Technology Challenges Exploring Solutions, Proof of Concept, VI Project Approval PART 2: The Present Day Current VI Deployment Current VI Projects Current VI Proof of Concepts Lessons Learned PART 3: Beyond Boundaries: The future of VI at IXIS DR, VDI and GRID Geographically Dispersed Virtual Datacenter Q & A Session
Company Profile – Global Operations IXIS Capital Markets is the U.S. subsidiary of IXIS Corporate & Investment Bank and a member of Paris-based Groupe Caisse d’Epargne 1.3 billion euros in 2005 2000 employees world wide
Company Profile – North American Operations Based in New York City, with branch offices in New Jersey, Los Angeles, Chicago, and Atlanta IXIS Capital Markets Products and Services include: Financing and Asset Securitization Securities and Derivatives Sales and Trading Asset and Portfolio Protection Products Investment Products Approximately 524 million (421 million euros) in revenues in 2005 Approximately 450 employees in North America and growing Small cohesive virtualization team with skills ranging across: Networking SAN Client/Server computing Virtual Infrastructure
IXIS Technology Challenges in 2004 Cost Control Annual Physical server deployment costs Server hardware costs (number of servers * cost per server) Network and SAN Infrastructure costs (number of ports * cost per port) Labor costs (number of hours to provision new server * hourly rate) Risk Control Maintenance Downtime Impact Hardware failures and upgrades Patching and upgrades of OS and Application Building mandated preventative maintenance New York Production Datacenter Limitations UPS HVAC Space constraints
Exploring Solutions With / Without Virtualization Infrastructure Private Host OS and VM control Reduction in space, power use and cooling Reduction in Network and SAN Infrastructure Consolidated hardware management and KVM Consolidation of applications on existing servers NO NO POOR NO NO Consolidation of applications on existing servers YES YES FAIR FAIR NO Consolidation of applications on blade servers YES YES EXCELLENT EXCELLENT YES Consolidation combining VMware ESX Server and IBM Blade Servers
Proof of Concept Proof of Concept (POC) Compiled a well defined scope for the POC Validation of ESX and VirtualCenter Validation of IBM Blade server platform Collaboration with IXIS IT Management Reviewed the scope of the POC to ensure alignment with the business drivers for the project Collaboration with external vendors (VMware and IBM) Reviewed the scope of the POC and business drivers for the project Created a focused SWAT team with the right skill set for the POC Networking SAN Client/Server computing
Proof of Concept IXIS POC Report Card Create a base Windows 2003 Server template via VC Deploy 6 new VMs from base template via VC Deploy Citrix Infrastructure on VMs Perform VMotion operations on VMs within IBM chassis Perform VMotion operations on VMs across two IBM chassis Perform a P2V operation Simulated IBM Bladecenter power supply failure testing Failover testing of the IBM Bladecenter management modules Failover testing of the IBM Bladecenter Network switches Failover testing of the IBM Bladecenter SAN switches PASS/FAIL
Proof of Concept and VI Project Approval Citrix Infrastructure (Secure Gateway, Secure Ticket Authority and 4 x Citrix servers)
Current VI Deployment SYS TEM PWR MGM T ACTIVE RE SET STATUS PWR MGM T RE SET WS-X6K-S2-MSFC2 1 S WITCH LOA D P ORT 1 100% SUPE RV ISOR 2 CONSOLE CONS OLE PORT MODE CONSOLE CONS OLE PORT MODE 1% P CMCIA PORT 2 LINK LINK E JECT SYS TEM ACTIVE STATUS WS-X6K-S2-MSFC2 2 S WITCH LOA D P ORT 1 100% SUPE RV ISOR 2 1% P CMCIA PORT 2 LINK LINK E JECT WS-X6416-GB IC 1 3 5 7 9 11 13 15 3 2 S TATUS 4 LI NK LI NK 16 P ORT GI GABI T E THE RNET 1 6 LI NK 2 LINK 3 8 LINK 4 10 LINK 5 LI NK 6 LINK 7 12 LINK 8 14 LINK LI NK 9 10 LI NK 16 LINK 11 12 LINK LI NK 13 14 LI NK 15 16 WS-X6416-GB IC 1 3 5 7 9 11 13 2 4 6 8 10 12 14 15 4 S TATUS LI NK LI NK 16 P ORT GI GABI T E THE RNET WS-X6548-RJ -45 1 1 2 LI NK 2 LINK 3 3 4 5 6 7 8 LINK 4 LINK 5 9 10 11 12 13 14 LI NK 6 LINK 7 15 16 17 18 19 20 LINK 8 LINK LI NK 9 10 21 22 23 24 25 26 LI NK 16 LINK 11 12 27 28 29 30 31 32 LINK LI NK 13 14 33 34 35 36 37 38 LI NK 15 16 39 40 41 42 43 44 45 46 47 48 5 STAT US PHONE 1 2 3 2 3 4 5 4 5 6 7 6 7 8 9 8 9 10 11 10 11 12 13 12 14 15 16 17 13 14 15 16 13 14 15 16 17 13 14 15 16 13 14 15 16 13 14 15 16 13 14 15 16 18 19 18 19 20 21 22 23 24 22 23 24 25 26 27 28 29 30 31 28 29 30 31 32 33 34 32 33 35 36 37 38 36 37 39 40 41 40 41 42 43 44 45 46 43 44 45 46 42 43 44 45 46 43 44 45 46 43 44 45 46 43 44 45 46 43 44 45 46 47 48 6 WS-X6148-RJ -45V 1 2 3 4 5 6 7 8 9 10 11 12 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 47 48 7 STATUS PHONE 1 WS-X6148-RJ -45V 1 2 3 4 5 6 7 8 9 10 11 12 17 18 20 21 19 20 21 22 25 23 24 26 27 25 26 27 28 29 30 31 32 34 35 33 34 35 36 38 39 37 38 39 40 41 42 47 48 47 48 8 STATUS PHONE FAN STATUS 1 WS-X6148-RJ -45V 2 3 4 1 2 5 3 4 6 7 5 6 8 9 7 8 10 11 9 10 12 11 12 17 18 19 17 18 20 21 19 20 22 23 24 21 22 25 23 24 26 27 25 26 28 29 30 27 28 31 29 30 32 33 31 32 34 35 33 34 36 37 35 36 38 39 37 38 40 41 39 40 42 41 42 47 48 47 48 9 STATUS PHONE 1 2 3 4 5 6 7 8 9 10 11 12 17 18 19 20 21 22 200-240 V 23 A 60/5 0 Hz 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 47 48 200-240 V 23 A 60/5 0 Hz T ALL INS RUN T ALL INS RUN INPUT OK FAN OK OUTPUT FAIL INPUT OK Power Supply 1 FAN OK OUTPUT FAIL Power Supply 2 Catalyst 6500 SERIES SYS TEM PWR MGM T ACTIVE S WITCH LOA D RE SET STATUS WS-X6K-S2-MSFC2 1 P ORT 1 100% SUPE RV ISOR 2 CONSOLE CONS OLE PORT MODE CONSOLE CONS OLE PORT MODE 1% P CMCIA PORT 2 LINK LINK E JECT SYS TEM PWR MGM T ACTIVE S WITCH LOA D RE SET STATUS WS-X6K-S2-MSFC2 2 P ORT 1 100% SUPE RV ISOR 2 1% P CMCIA PORT 2 LINK LINK E JECT WS-X6416-GB IC 1 3 5 7 9 11 13 15 3 2 S TATUS 4 LI NK LI NK 16 P ORT GI GABI T E THE RNET 1 6 LI NK 2 LINK 3 8 LINK 4 10 LINK 5 LI NK 6 LINK 7 12 LINK 8 14 LINK LI NK 9 10 LI NK 16 LINK 11 12 LINK LI NK 13 14 LI NK 15 16 WS-X6416-GB IC 1 3 5 7 9 11 13 15 4 2 S TATUS 4 LI NK LI NK 16 P ORT GI GABI T E THE RNET WS-X6548-RJ -45 1 1 2 6 LI NK 2 LINK 3 3 4 5 6 7 8 8 LINK 4 11 12 10 LINK 5 9 10 13 14 LI NK 6 LINK 7 15 16 17 18 19 20 12 LINK 8 14 LINK LI NK 9 10 21 22 23 24 25 26 LI NK 16 LINK 11 12 27 28 29 30 31 32 LINK LI NK 13 14 33 34 35 36 37 38 LI NK 15 16 39 40 41 42 43 44 45 46 47 48 5 STAT US PHONE 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 6 WS-X6148-RJ -45V 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 13 14 15 16 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 43 44 45 46 43 44 45 46 47 48 7 STATUS PHONE 1 WS-X6148-RJ -45V 2 3 1 2 4 5 3 4 6 7 5 6 8 9 7 8 10 11 9 10 12 11 12 17 18 19 17 18 20 21 19 20 22 23 24 21 22 25 23 24 26 27 25 26 28 27 28 29 30 29 30 31 32 33 31 32 34 35 33 34 36 35 36 37 38 39 37 38 40 41 39 40 42 41 42 47 48 47 48 8 STATUS PHONE FAN STATUS 1 WS-X6148-RJ -45V 2 3 2 3 1 2 4 5 4 5 3 4 6 7 6 7 5 6 8 9 8 9 7 8 10 11 10 11 9 10 12 13 14 13 14 15 16 12 13 14 15 16 11 12 15 16 17 18 19 18 19 17 18 20 21 19 20 22 23 24 21 22 25 23 24 26 27 26 27 25 26 28 27 28 29 30 29 30 31 32 33 32 33 31 32 34 35 34 35 33 34 36 35 36 37 38 39 38 39 37 38 40 41 40 41 39 40 42 43 44 43 44 45 46 42 43 44 45 46 41 42 45 46 47 48 47 48 9 STATUS PHONE 1 200-240 V 23 A 60/5 0 Hz 17 20 21 22 23 24 25 28 29 30 31 36 37 47 48 200-240 V 23 A 60/5 0 Hz T ALL INS RUN T ALL INS RUN INPUT OK FAN OK OUTPUT FAIL INPUT OK Power Supply 1 FAN OK OUTPUT FAIL Power Supply 2 Catalyst 6500 SERIES VirtualCenter version ESX servers ESX version VMFS partitions VMFS partition size Active VMs NY (PROD) NJ (PROD / DR) NJ (PROD / DR) 1.2 28 2.5.1 build 14182 4 1 TB 94 1.3 14 2.5.3 build 22981 3 1 TB 53 2.0.1 18 3.0.1 build 32039 3 1 TB 53
Current VI Deployment Power Supply Overview Each PSU from a chassis connects to a separate PDU, providing protection against: 30 Amp electrical circuit failure
Current VI Deployment CONSOLE PORT MODE CONSOLE CONSOLE PORT MODE PWR MGMT SUPERVISOR 2 SUPERVISOR 2 CONSOLE CONSOLE PORT MODE CONSOLE CONSOLE PORT MODE PORT 1 EJECT PCMCIA EJECT PORT 2 K LIN K LIN 1% PCMCIA PWR MGMT SWITCH LOAD PORT 1 100% K LIN K LIN 1% PCMCIA RESET 2 SYS TE M PORT 2 ACTIVE PORT 1 100% STATUS PWR MGMT SWITCH LOAD RESET SYS TE M RESET EJECT WS-X6K-S2-MSFC2 ACTIVE STATUS SWITCH LOAD 100% K LIN K LIN 1% PCMCIA SYS TE M PORT 2 1 CONSOLE ACTIVE PORT 1 100% SUPERVISOR 2 WS-X6K-S2-MSFC2 2 STATUS PWR MGMT SWITCH LOAD RESET SYS TE M WS-X6K-S2-MSFC2 ACTIVE 1 STATUS WS-X6K-S2-MSFC2 Network Overview Two CISCO Gigabit Network switches per chassis Two Trunks per CISCO Gigabit Network switch One Trunk per CISCO Gigabit switch connected to alternate 6509 Core switches, providing protection against: CISCO Gigabit Network switch, trunk or port failures within the IBM Chassis A CISCO 6509 Core, trunk or port failure Port Groups configured on ESX server to service ESX Console, VMotion operations and VMs EJECT SUPERVISOR 2 PORT 2 K LIN K LIN 1% WS-X6416-GBIC WS-X6416-GBIC 1 3 5 7 9 11 13 1 15 3 3 5 7 9 11 13 15 3 2 STATUS 4 LINK 16 PORT GIGABIT ETHERNET LINK 1 6 LINK 2 LINK 3 8 LINK 4 LINK 5 10 LINK 6 LINK 7 12 LINK 8 LINK 14 LINK 9 10 LINK 16 LINK 11 12 LINK 2 STATUS LINK 13 14 LINK 4 LINK 16 PORT GIGABIT ETHERNET 15 16 LINK 1 2 1 2 6 LINK LINK 3 4 3 4 8 LINK LINK 5 6 5 6 10 LINK LINK 7 8 7 8 12 LINK LINK 14 LINK 9 10 LINK 16 LINK 11 12 LINK LINK 13 14 LINK 15 16 WS-X6416-GBIC WS-X6416-GBIC 1 3 5 7 9 11 13 1 15 4 3 5 7 9 11 13 15 4 2 STATUS 4 LINK 16 PORT GIGABIT ETHERNET WS-X6548-RJ-45 LINK 1 1 2 6 LINK 2 LINK 3 3 4 5 6 7 8 8 LINK 4 9 10 LINK 5 11 12 13 14 10 LINK 6 15 16 LINK 7 17 18 19 20 12 LINK 8 LINK 14 LINK 9 10 21 22 23 24 25 26 27 28 LINK 16 LINK 11 12 29 30 31 32 33 34 LINK 35 36 37 38 39 40 LINK 43 44 45 46 4 LINK 16 PORT GIGABIT ETHERNET 15 16 41 42 2 STATUS LINK 13 14 WS-X6548-RJ-45 47 48 5 1 2 LINK 3 4 6 LINK 5 6 7 8 LINK 9 10 8 LINK LINK 10 LINK LINK 12 LINK LINK 14 LINK 9 10 LINK 16 LINK 11 12 LINK LINK 13 14 LINK 15 16 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 5 STATUS STATUS PHONE 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 PHONE 47 48 1 6 2 3 4 5 6 7 8 9 10 6 WS-X6148-RJ-45V 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 WS-X6148-RJ-45V 7 1 2 3 4 5 6 7 8 9 10 7 STATUS STATUS PHONE 1 WS-X6148-RJ-45V 2 3 2 3 1 2 4 5 4 5 3 4 6 7 6 7 5 6 8 9 8 9 7 8 10 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 PHONE 1 WS-X6148-RJ-45V 8 2 3 2 3 1 2 4 5 4 5 3 4 6 7 6 7 5 6 8 9 8 9 7 8 10 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 8 STATUS STATUS PHONE FAN STATUS 1 WS-X6148-RJ-45V 1 2 3 4 5 6 7 8 10 9 10 PHONE FAN STATUS 1 WS-X6148-RJ-45V 9 1 2 3 4 5 6 7 8 10 9 10 9 STATUS STATUS PHONE 1 2 3 4 5 6 7 8 9 10 200-240 V 23 A 60/50 Hz 200-240 V 23 A 60/50 Hz INPUT OK FAN OUTPUT OK FAIL FAN OUTPUT OK FAIL Power Supply 2 Catalyst 6500 SERIES 3 4 5 6 7 8 9 10 200-240 V 23 A 60/50 Hz L AL ST N IN RU L AL ST N IN RU INPUT OK Power Supply 1 2 200-240 V 23 A 60/50 Hz L AL ST N IN RU L AL ST N IN RU PHONE 1 INPUT OK FAN OUTPUT OK FAIL INPUT OK Power Supply 1 FAN OUTPUT OK FAIL Power Supply 2 Catalyst 6500 SERIES
Current VI Deployment CHASSIS 1 Blades 1 to 7 Storage Overview Two McDATA SAN switches per chassis Two ISLs per McDATA SAN switch Each McDATA SAN switch is mapped directly to a single Connectrix Director Class switch, providing protection against: McDATA SAN switch, ISL or port failures within the IBM Chassis Connectrix Director Class switch, ISL or port failure 14 ESX Hosts mapped to a VMFS LUN VMotion operations possible: Within a chassis Across a chassis VMs 1 TB VMFS LUN 335 GB VMFS LUN 1 TB VMFS LUN VM Templates Roll Back VMs CHASSIS 2 Blades 8 to 14 VMs
Current VI Deployment IXIS VI 2 Report Card Server Consolidation Increased host resource utilization Zero Downtime Upgrades with VMotion Datacenter relocation with no moving trucks Increased server to administrator ratio Reduced server deployment times Infrastructure cost savings Labor cost savings Centralized management of geographically dispersed Virtual Infrastructure PASS/FAIL
Current VI Projects VI 3 Deployment Virtualization Features /Support Symmetric Multiple Processing (SMP) Memory ESX Host to LUN mapping 64 Bit Support VMotion Distributed Services (RP HA, DRS and VCB) VI 2 2 Way 3.6 GB 16 No Yes No VI 3 4 Way 16 GB 32 Yes Yes Yes Workloads / services currently virtualized on VI 2 DCs, Printing, IIS, Avocent DSView, CAVA, Citrix,Documentum, Datasynapse Engines, Tradeweb, SLX, KVS, SQL, Oracle Workloads / services not yet virtualized Compute Intensive (PolyPaths) and MS Exchange
Current VI Projects VI 3 Storage and Distributed Services 28 ESX Hosts mapped to a LUN VMotion operations possible: Within a chassis Across a chassis VI 3 Clusters and Resource Pools (RP) Utility computing model Define RP per service DRS Automated VMotion, optimized resource allocation HA Automated restart, improved service uptimes
Current VI Proof of Concepts Application GRID on VI 2 In production and proving to be a success Application GRID on VI 3 Optimization at the application layer with the GRID application Optimization at the hardware layer with DRS and HA Resulting in a highly optimized fault tolerant GRID computing environment
Current VI Proof of Concepts Virtual Desktop Infrastructure (VDI) Small POC underway POC users do not require dual head video display Benefits of moving to VDI Cost reductions Improved provisioning of desktops User desktop services hosted on a fault tolerant infrastructure Enhanced Disaster Recovery
Lessons Learned Architect your VI carefully Network and SAN are key components of the VI Construct a VI roadmap Project ESX and VM totals TAM service adds critical value Single point of accountability Not all workloads should be virtualized Users and workloads differ by environment, consider this carefully Spread the Virtualization story internally to all of members of IT Virtualization for most is still a new and unknown technology
Beyond Boundaries - The Future of VI at IXIS Enhanced Disaster Recovery Solutions Replication of VMFS or individual VMs between sites Manual / Auto restart of replicated VMs at time of test or disaster Centralized management of the distributed VI Enhanced VDI Solutions Leverage distributed replicated VI for VDI at time of test or disaster Support for multi video head virtual desktops Enhanced Application GRID Geographically dispersed Application GRID
Beyond Boundaries - The Future of VI at IXIS Geographically Dispersed Virtual Datacenter DATACENTER #1 HA MANAGEMENT HARDWARE VIRTUALIZATION DATACENTER #2 DRS DRS VirtualCenter Management Server VirtualCenter Management Server VMs VMs MANAGEMENT apps apps apps apps GRID apps apps apps apps GRID XP Pro RHEL W2K3 W2K3 OS XP Pro RHEL W2K3 W2K3 OS Virtual SMP ESX Servers VMFS VMFS VMs ENTERPRISE STORAGE HA VMFS LUN ESX Servers Virtual SMP HARDWARE VIRTUALIZATION VMs VMFS LUN VMFS LUN ENTERPRISE NETWORK VMFS LUN ENTERPRISE STORAGE ENTERPRISE NETWORK WAN CONNECTIVITY
Session Summary Reality check, datacenters are changing from physical to virtual, so be prepared Architect your VI carefully, SAN and Network are key components Develop a Virtual Infrastructure Roadmap VI 3 will enable a utility computing model Integration of VI into enhanced DR solutions
QUESTIONS The Always-On Enterprise: Acheiving Higher Service Levels with VMware Mornay Van Der Walt Vice President, System Architect IXIS Capital Markets
Backup Slides – VI Components Virtual Infrastructure (VI) Physical components Enterprise Servers Enterprise Network Enterprise Storage Virtualization components ESX servers Virtual Network VMFS Virtual Machine (VM) Management component Virtual Center
Backup Slides - Increased Host Resource Utilization
Presentation Download Please remember to complete your session evaluation form and return it to the room monitors as you exit the session The presentation for this session can be downloaded at http://www.vmware.com/vmtn/vmworld/sessions/ Enter the following to download (case-sensitive): Username: cbv rep Password: cbvfor9v9r
ws-x6k-s2-msfc2 status system active pwr mgm t re set console port 1 port 2 pcmcia eject mode port switch load 1% 100% li nk l n supervisor 2 ws-x6k-s2-msfc2 status system active pwr mgm t reset console port 1 port 2 pcmcia eject mode port switch load 1% 100% li nk l n status ws-x6416-gbic 16 port gigabit ethernet link link 1 2 12 li nk li nk 15 16
May 02, 2018 · D. Program Evaluation ͟The organization has provided a description of the framework for how each program will be evaluated. The framework should include all the elements below: ͟The evaluation methods are cost-effective for the organization ͟Quantitative and qualitative data is being collected (at Basics tier, data collection must have begun)
Silat is a combative art of self-defense and survival rooted from Matay archipelago. It was traced at thé early of Langkasuka Kingdom (2nd century CE) till thé reign of Melaka (Malaysia) Sultanate era (13th century). Silat has now evolved to become part of social culture and tradition with thé appearance of a fine physical and spiritual .
On an exceptional basis, Member States may request UNESCO to provide thé candidates with access to thé platform so they can complète thé form by themselves. Thèse requests must be addressed to esd rize unesco. or by 15 A ril 2021 UNESCO will provide thé nomineewith accessto thé platform via their émail address.
̶The leading indicator of employee engagement is based on the quality of the relationship between employee and supervisor Empower your managers! ̶Help them understand the impact on the organization ̶Share important changes, plan options, tasks, and deadlines ̶Provide key messages and talking points ̶Prepare them to answer employee questions
Dr. Sunita Bharatwal** Dr. Pawan Garga*** Abstract Customer satisfaction is derived from thè functionalities and values, a product or Service can provide. The current study aims to segregate thè dimensions of ordine Service quality and gather insights on its impact on web shopping. The trends of purchases have
Chính Văn.- Còn đức Thế tôn thì tuệ giác cực kỳ trong sạch 8: hiện hành bất nhị 9, đạt đến vô tướng 10, đứng vào chỗ đứng của các đức Thế tôn 11, thể hiện tính bình đẳng của các Ngài, đến chỗ không còn chướng ngại 12, giáo pháp không thể khuynh đảo, tâm thức không bị cản trở, cái được
Le genou de Lucy. Odile Jacob. 1999. Coppens Y. Pré-textes. L’homme préhistorique en morceaux. Eds Odile Jacob. 2011. Costentin J., Delaveau P. Café, thé, chocolat, les bons effets sur le cerveau et pour le corps. Editions Odile Jacob. 2010. Crawford M., Marsh D. The driving force : food in human evolution and the future.
Le genou de Lucy. Odile Jacob. 1999. Coppens Y. Pré-textes. L’homme préhistorique en morceaux. Eds Odile Jacob. 2011. Costentin J., Delaveau P. Café, thé, chocolat, les bons effets sur le cerveau et pour le corps. Editions Odile Jacob. 2010. 3 Crawford M., Marsh D. The driving force : food in human evolution and the future.
