www.it-ebooks.info BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation Master one of the world's most powerful enterprise workload automation tools – BMC Control-M 7 - using this book Qiang Ding BIRMINGHAM - MUMBAI www.it-ebooks.info BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation Copyright © 2012 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. First published: October 2012 Production Reference: 1041012 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-84968-256-5 www.packtpub.com Cover Image by Anvar Khodzhaev (cbetah@yahoo.com) www.it-ebooks.info Credits Author Qiang Ding Reviewers Bentze Perlmutter Robert Stinnett Acquisition Editor Dhwani Devater Technical Editor Lubna Shaikh Copy Editors Brandt D'mello Insiya Morbiwala Laxmi Subramanian Project Coordinator Vishal Bodwani Proofreaders Lesley Harrison Lynda Sliwoski Indexer Rekha Nair Graphics Nilesh R. Mohite Manu Joseph Production Coordinator Arvindkumar Gupta Cover Work Arvindkumar Gupta www.it-ebooks.info About the Author Qiang Ding (Melbourne, Australia) has been working within the Control-M space for more than a quarter of his life. During his early days at BMC Software, Qiang resolved countless number of critical technical issues for Control-M customers around the world from Fortune 500 companies to government organizations. In recent years, Qiang has travelled hundreds thousands of miles around Australia and the North AP area to help many organizations to design, manage, and optimize their batch workload automation environment and to extend his passion to others by delivering Control-M trainings to end users and BMC Partners. Currently Qiang is temporary living in Sydney and working on a enterprise wide Control-M migration and Consolidation project for a major Australian bank. He enjoys working with other experts in the eld and is constantly involved in nding ways for making improvements to the batch environment that he works on. www.it-ebooks.info Acknowledgement There are many people that I would like to thank for their contribution to the creation of this book and to those who have reviewed, proofread, commented, and provided quotes. On a greater scale, I would like to thank Bentze Perlmutter and Graeme Byrnes, who originally taught me every technical detail of Control-M, followed by Bruce Roberts, who in the recent years embraced my Control-M knowledge from a pure technical level into business level. I also would like to thank people who I worked for and worked with in the past few years, those who had a faith in me and gave me the opportunity and trust, including Allen Lee, Amy You, Angel Wong, Bao Ling, Louis Cimiotti, Chris Cunningham, Craig Taprell, Curtis Eddington, David Timms, Digby Pritchard, Doug Vail, Ian Jones, Jason St. Clair, Jeffrey Merriel, Jim Darragh, Matthew Sun, Min Yuan, Moshe Miller, Rabin Sarkar, Rick Brown, Shaun Kimpton, Stephen Donnelly, Tom Geva, Tristan Gutsche, Xianhua Peng, Yuan Yuan, and Ze'ev Gross. Last but not least, I would like to thank my friend Mike Palmer who inspired me and guided me in every aspect of my life. For all those who have provided support and guidance over the years and if I have yet to mention your name, I sincerely apologize. www.it-ebooks.info About the Reviewers Bentze Perlmutter has 15 years of IT experience, working in various companies and holding positions in operations, QA, technical support, engineering, systems management, and production control. His main area of expertise is batch and workload automation using tools such as Control-M and AutoSys. He has worked on complex projects requiring evaluation of solutions, design and planning, implementation, administration, and on-going support within large organizations, mainly in the nancial industry. Robert Stinnett has worked with automation systems on various platforms from mainframe to distributed since 1992. He has been using Control-M since 2003 when he was the lead for bringing it to CARFAX, the leading provider and pioneer of vehicle history reports, where he has worked for the past 10 years. Robert is active in many Control-M communities, has given presentations at various conferences on the capabilities and cost-benets of using an automated workload management platform, and has written a number of open-source utilities to help take advantage of and extend Control-M's capabilities on the distributed side. One of the next big things he sees for automation systems is their integration with Cloud. He is currently working on projects that explore how Cloud can be used for providing capacity on demand as well as providing redundancy and scalability to existing automation and scheduling implementations. Robert is also an active member of the Computer Measurement Group where he sees the role of automation in IT as one of the major reasons to have a sound performance and capacity management program in place to help manage the continuing technological evolution taking place in businesses. He can be reached at Robert@robertstinnett.com. www.it-ebooks.info www.PacktPub.com Support les, eBooks, discount offers and more You might want to visit www.PacktPub.com for support les and downloads related to your book. Did you know that Packt offers eBook versions of every book published, with PDF and ePub les available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at service@packtpub.com for more details. At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks. http://PacktLib.PacktPub.com Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can access, read and search across Packt's entire library of books. Why Subscribe? • Fully searchable across every book published by Packt • Copy and paste, print and bookmark content • On demand and accessible via web browser Free Access for Packt account holders If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate access. Instant Updates on New Packt Books Get notied! Find out when new books are published by following @PacktEnterprise on Twitter, or the Packt Enterprise Facebook page. www.it-ebooks.info www.it-ebooks.info Table of Contents Preface 1 Chapter 1: Get to Know the Concept 7 Introduce batch processing 8 The history of batch processing 8 Batch processing versus interactive processing 11 Time-based batch- and event-driven batch 13 Is this the end for batch processing? 14 Running batch processing tasks 16 Automating batch processing 21 Basic elements of a job 21 What to trigger 22 When to trigger (Job's scheduling criteria) 22 Dependencies (Job's predecessors and dependents) 22 More advanced features of scheduling tools 24 Ability to generate notications for specied events 24 Ability to handle an external event-driven batch 24 Intelligent scheduling – decision-making based on predened conditions 25 Security features 25 Additional reporting, auditing, and history tracking features 26 Centralized enterprise scheduling 26 Challenges in today's batch processing 26 Processing time 31 Batch window length 31 Batch monitoring and management 32 Cross-time zone scheduling 32 Resource utilization 32 Maintenance and troubleshooting 33 Reporting 33 Reacting to changes 33 The solution 34 Processing time and resource utilization 35 Batch monitoring and management 35 www.it-ebooks.info [...]...Table of Contents Cross-time zone scheduling 36 Maintenance and troubleshooting 37 Reporting 37 Reacting to changes 37 From batch scheduling to workload automation Batch scheduling: Static scheduling The Workload Automation concept Dynamic batch processing with virtualization technology and Cloud computing Integration with real-time system, workload reusability Summary Chapter 2: Exploring Control-M. .. technical details of the tool - How to plan, install, use, as well as manage a Control-M environment, and finally look at how to leavage the tool to meet the already sophiscated and ever growing business demand Throughout the book, you will learn important concepts and features of Control-M through detailed explainations and examples, as well as learn from the author's experience, accumulated over many... shop's order to suppliers • 3:00am: As all outstanding orders have been processed, a backup job is started for backing up the database and filesystem • 5:00am: A batch job generates and sends yesterday's sales report to the accounting department • 5:15am: Another batch job generates and sends a stock on hand report to the warehouse and purchasing department • 5:30am: A script gets triggered to clean up old... Applying Control-M fix packs and patches 280 Installing additional Control-M GUI clients Frequent administration tasks Stop/start components 285 288 288 Defining additional GUI users and groups 298 When to apply fix packs and patches How to apply fix packs and patches Fix pack and patch installations in our environment Manually stop/start components Configuring automatic startup script Authorization of... challenges at each stage It also talks about the sizing and the technical considerations that are necessary for building a solid batch infrastructure Finally, it shows how to get started into the technical details and prepare machines and the environment for the Control-M implementation Chapter 4, Creating and Managing Batch Flows with Control-M GUI, looks at the important job scheduling concepts of Control-M. .. such as sales conciliation or reporting In addition, because during the night time there are fewer changes to the data, batch jobs can have more freedom when manipulating the data and it allows the system to perform the backup tasks [ 12 ] www.it-ebooks.info Chapter 1 Time-based batch- and event-driven batch What we have discussed so far – Batch processing defined to run during a certain time is traditional. .. "Start" job Summary Chapter 5: Administrating the Control-M Infrastructure Additional component installations Installation of BIM and Forecast Installation Post-installation tasks Configuring BIM web interface Installation of Control Modules Pre-installation considerations Installation – Control-M for database Installation – Control Module for Advanced File Transfer Installation – Control-M Business... Control-M/ EM: MaxOldDay and MaxOldTotal Control-M/ EM: Default AverageTime Control-M/ Server: New Day Time Control-M/ Server: Simple Mail Transfer Protocol parameters Control-M/ Server: shout destination tables Summary Chapter 8: Road to Workload Automation Integrating Control-M with business processes Building the environment Interacting with BPI interfaces Technical background Defining an account Triggering... requirements and balance the system utilization As a fact, large orgnisations today are still investing in batch processing and continuously trying to figure out how to make it run better Running batch processing tasks To understand batch processing further we need to begin from it's father - the mainframe computers Job Control Language (JCL) was introduced as a basic tool on the mainframe computers... the Control-M/ EM GUI client to discover more GUI features, such as advanced functionalities offered in ViewPoints and archived ViewPoints It also gives an overview of how to use BIM and Forecast to proactively monitor jobs and estimate potential impacts, by creating What-if scenarios Towards the end, it visits the Reporting Facility, takes a look at each available report type, and discusses how to automate . around Australia and the North AP area to help many organizations to design, manage, and optimize their batch workload automation environment and to extend. www.it-ebooks.info BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation Master one of the world's most powerful enterprise workload