Project description (functionalities and uses):
The hunan mobile communications corporation operations support system (boss system) is a core business support system that integrates operations, billing, clearing, accounting, charging, etc., and achieves the operational principles of “client-centred, business-oriented development and management, and business-oriented network management” and significantly enhances the marketing and service levels of enterprises。
Project dilemmas and solutions:
The boss system has been on line since early 2003 and performance problems have arisen since the start-up of the system. The main manifestation is that interactive responses to end-users are less than expected, especially when operations are busy, and are less timely. The observation from the mainframe (aix) system is mainly reflected in the system's i/o wait. The accounting system consists of several components stored by operational applications, the oracle database, the aix mainframe and ibm ess, making the positioning and optimization of performance bottlenecks more complex。
The difficulties of the project are mainly as follows:
1 hunan mobile communications boss system is a large complex system. The system includes the following levels: application, database, mainframe system (operating system), san network and eess storage system. In cases of systemic performance problems, the positioning and optimization of performance issues are complex。
2 the database is large, with an overall data capacity of about 2120 gb, and the entire data migration takes dozens of hours, while the production system does not allow for long shutdown times。
3 hunan mobile boss is an application of 7x24 and no stoppage is permitted。
4 the application of the sql statements in the section is complex and the inappropriate use of sql statements, such as writing and verbs, directly results in inefficient sql operations, occupancy of systems i/o and memory, which need to be identified and adjusted。
5 some of the parameters of the oracle database need to be adjusted。
The nuclear parameters within the operating system need to be adjusted。
7 the physical distribution of database data on eds storage needs to be re-established
The following solutions have been adopted for each of these issues:
1 in response to performance problems arising from the hunan mobile boss system system, based on the practical application of the hunan mobile boss system system, the mainframe system and storage were monitored, adjusted and optimized using performance monitoring tools such as the ess express and precise software, and the oracle database and application system was streamlined。
The project is divided into the following phases:
# check all hardware systems in the boss system system, especially in the san network。
• check the data flow of the san switch and see if there is any problem of asymmetric traffic, loss of data packages or error in data transmission。
• analyse the data distribution on ess, install and configure the ess express monitoring software, and observe the existence of uneven loads of fc channels, clusters, sa cards or sa loop。
# check and optimize the parameters of aix running on the mainframe system to fit the system system。
♪ install and configure precise indepth for oracle, check the parameters set in oracle database, identify the most influential applications and assist software developers in optimizing applications。
♪ two adjustments to the data distribution on the ess storage system and monitoring the performance of the ess storage system through the stonewatch expert software
# use precise software to monitor the use of system resources in databases and applications, further upgrade the mainframe system and make recommendations on the upgrading of oracle databases and applications
♪ the location of performance bottlenecks
The general strategy for harmonization is as follows:
In the condensation of the south movement, the ibm storage and mainframe systems are first subjected to in-depth parameter and configuration adjustments, provided that the design and application of the database are not modified. At the same time, with the full cooperation of the technical staff of the hunan mobile billing centre, large-scale adjustments were made to the distribution of data on the eess storage system, and monitoring of the performance parameters of the oracle database through precise software, positioning of severe bottlenecks in the consumption of resources such as cpu, i/o, and providing performance optimization advice to the oracle database and application system。
2. Modify oracle performance parameters for the performance status of applications。
Cursor sharing's value changed from exact to value, reduced
The value of spin count was adjusted from 2000 to 5,000.
# on the basis of precise monitoring and analysis, check the logical design of the sql statement that consumes the most resources, store the table data that ranks against the former sql statement separately from the index, establish an appropriate partition index and increase the parallelity between the former sql statement on resource consumption。
# continuous monitoring of the use of ess through the starwatch express software to capture the performance and use of ess。
# through the analysis of oracle and limited data in precise software, it is determined that the current application is not in the desired state of operation, and it is recommended that the application be checked and adjusted accordingly. At the same time, in order to better identify application problems, it is recommended that data be collected for longer periods of time before further analysis is undertaken。
3 adjust data distribution on ess。
First, the data are distributed evenly in two clusters, then in as many corridors as possible。
Since the total data capacity is approximately 2120gb, the total data migration takes dozens of hours, while the production system does not allow for long shutdown times。
Based on the validation of multiple programmes, it was decided to implement data migration using logical mirrors. The concrete step is to create mirror images of all the logical rolls on the purpose hard drive, to synchronize the data, and to delete the mirror section on the original hard drive. The entire data migration was carried out in the back of the system for a total of 60 hours and all data migrations were completed。
4. Adjustment of nuclear parameters in the operating system
5 check the logical design of the maximum resource consumption statement。
Create an appropriate partition index
Stores the ranking by the table data of the foreword and the index。
Increasing the parallelity of resource consumption with the preceding sentence。
Oracle's argument cursor sharing is set to decrease force
Adjusting the number of latchs (e. G. Db blonk lru latches) or some memory parameters (e. G. Shared pol size) solves the internal lock problem。
6 as a result of the data storage adjustments, the system i/o is in good performance, performance at the system and storage end has been optimized and performance at the system and storage end has been optimized. It is recommended that the next focus of the client's work be to address the performance bottlenecks of the application system, to check the logical design of the application and the establishment and distribution of the data index, taking into account the results of precise's monitoring of the application. Assist application developers in adaptation。
7 adjustment of operating system exchange space (page space)。
A summary of project successes and failures:
The success of the project in the performance of the mobile bos system in hunan has been as follows:
1. The sound structure design (physical and logical) of the database for the preceding period is key to the success or failure of the project as a whole, which provides a good basis for a stable and efficient operation of the project as a whole, while also serving a multiplier effect. The system's i/o bottlenecks are directly caused by the unreasonable distribution of databases in physical ess storage in hunan mobile boss。
2. Reasonable adjustments are made to the configuration parameters of the database and to the nuclear parameters of the operating system to ensure a reasonable distribution and application of the physical resources of the system (cpu, i/o, memory and network) and logical resources (naked equipment, documentation systems, etc.) based on the physical resources of the system。
3. The use of third-party monitoring software (e. G., stowatch express and precise) allows faster positioning performance bottlenecks and faster performance adjustments。
4. Examine the logical design of the sql statement, which consumes the maximum amount of resources, store the table data and the index of the top sql statement separately, establish an appropriate partition index and increase the parallelity of the former sql statement for resource consumption. Many performance bottlenecks are often caused by “bad” sqls。
5. There is a need for a good performance adjustment process and performance adjustment methodology step by step, step by step, step by step, step by step, step by step, scale by step, and know the final positioning performance bottlenecks。
6. To be good at teamwork, and throughout the performance adjustments, i am responsible for software adjustments, working with the technical staff of the hardware and network adjustments。
Performance adjustment is a global exercise involving application development, intermediates, databases, operating systems, storage, networks, etc. A clear understanding of the global information architecture is needed。
Summary: in the performance adjustments for the hunan mobile boss system, i was involved as ibm technical adviser, and was responsible for the adjustment of the parameters of the oracle database configuration; for completing performance monitoring at the end of the database and for improving performance; and for examining the logical design of the sql statement, which consumes the greatest amount of resources, by storing table data and indexing the top sql statement separately, establishing an appropriate index of partitions, and increasing the parallelity of resource consumption with the former sql statement. Responsible for locating performance bottlenecks (cpu, memory, i/o and network bottlenecks) at the operating system level and adjusting nuclear parameters within the operating system. Responsible for assisting developers with application development modifications and sql performance adjustments
Generate poster
