The History of SIGMETRICS
The following facts, stories, and anecdotes have been compiled by Jennifer Harper at the University of Calgary, with assistance and encouragement from SIG Chair Carey Williamson. Most of the material is drawn from "the box" handed down from one SIG executive to another over the years, as well as the online versions of PER, which someone patiently digitally scanned and posted over a decade ago. Thank you, thank you, thank you!
To view stories about a particular era in the history of SIGMETRICS, just click on that row to expand its contents. Click again to collapse.
If you have other (printable!) stories that you would like to see here, please get in touch with your SIG Chair.
- Today
-
∞ The recent explosion in the use of two-way media allowing users to upload content (e.g. YouTube, FaceBook, Twitter, Wikipedia, BitTorrent) and the concurrent rise of wireless technology has made performance evaluation an even more crucial issues. Improvements in performance may be measured in fractional percentages or microseconds, but when these improvements are applied to the movements of terabytes worth of data, they add up. It is likely that unless significant improvements are made to the physical infrastructure of the Internet, it will soon be unusable due to the sheer volume of information being moved at any given moment. Cloud-based services, power consumption, and green computing are also becoming prominent research topics. It's a good time to be a member of SIGMETRICS! ∞ Today, SIGMETRICS is a vibrant and active SIG with about 500 members. The primary interests are in the development and application of methodologies for computer systems performance evaluation. The primary benefits offered to SIGMETRICS members are: - Discounted registration fee for all SIGMETRICS sponsored/co-sponsored conferences and workshops, as well as "in cooperation" events
- Access to SIGMETRICS-related content in ACM Digital Library
- Quarterly newsletter "Performance Evaluation Review", which contains open-call research articles, themed issues on special topics, and abstracts from SIG-related workshops. PER is printed and mailed to members 4 times per year, with one of the issues dedicated to the annual SIGMETRICS conference proceedings.
- Eligibility for SIG awards (Rising Star, Achievement Award)
- Support letters for eligible members applying for advanced ACM membership grades (Senior Member, Distinguished Member, Fellow)
- Moderated mailing list for SIGMETRICS members used judiciously for SIG communications (e.g., awards, nominations, and news items)
- Moderated "performance" mailing list (performance@merlot.usc.edu) for bulk monthly communications to the broader performance evaluation community (e.g., conference announcements, calls for papers, awards, job postings, software tools, questions)
-
- 2000s
-
∞ Throughout the 2000s, there was an increasing emphasis on the use of "metaprograms" to model hypothetical systems/networks/hardware, or simulate/emulate existing ones for the purposes of measuring speed, reliability, loss, etc. This opens up entirely new areas of performance analysis, since not only are the systems themselves being measured and analyzed, but the programs used to simulate these systems virtually are also subject to analysis. ∞ Performance measurement and evaluation has become an increasingly difficult subject, considering that performance is no longer simply a matter of how fast the job gets done, how much data can be stored, and how much bandwidth there is to work with. Security, reliability, resilience, and usability are also important measures of performance, and with the increasing complexity of both individual computers and networks and the fact that most users are only minimally computer literate, other factors apart from sheer power and capacity may take precedence in evaluating a given system. Historical Notes: ∞ The First Workshop on Performance and Architecture of Web Servers (PAWS 2000) was held in conjunction with SIGMETRICS 2000 on June 17 and 18 in Santa Clara, CA. ∞ The First Workshop on Practical Aspects of Performance Analysis (PAPA 2002) was held in conjunction with SIGMETRICS 2002 on June 15 in Marina Del Rey, CA. It was jointly sponsored by SIGMETRICS and the Computer Measurement Group (CMG), with partial support from HP Labs. This workshop focused on how performance analysis techniques can help practitioners in the design and performance management of computer systems. SIGMETRICS member Mark Squillante is finally content that SIGMETRICS conferences are gender-neutral (MAMA and PAPA). ∞ In 2005, SIGMETRICS executive elections were held online for the first time! ∞ Also in 2005, in conjunction with SIGMETRICS 2005 in Banff, Canada, the First Workshop on Large Scale Network Interference (LSNI 2005) took place on June 6, 2005. ∞ The First Workshop on Hot Topics in METRICS (HotMetrics'08) was held in Annapolis, MD, on June 6th, 2008, in conjunction with SIGMETRICS 2008. For the first time, the SIGMETRICS 2008 program is 3 full days, with 36 papers presented in addition to workshop and poster papers. ∞ At SIGMETRICS 2009 in Seattle, WA, (June 15-19) the First GreenMetrics Workshop was held to discuss topics in performance evaluation, energy efficiency, and environmental sustainability. Anecdotes: ∞ "In recent years, large scale network inference has attracted significant interest within the research community. On one front, considerable progress has been made on traffic matrix estimation. Solutions have been proposed to estimate the amount of traffic flowing between any pair of ingress and egress points within an IP network simply based on the total amount of traffic recorded over IP links. On another front, efforts are being made to detect the state of the network from end to end measurements using inference techniques or to infer the traffic workload by exploiting application behavior. In essence, the full instrumentation of the state of an IP network is still considered a cost prohibitive task and inference may be the only tool we have to understand the behavior of such large scale systems. The potential benefits of the proposed estimation techniques can be great. Accurate measurement of an IP traffic matrix is essential for network design and planning. Moreover, accurate estimation of the network state can facilitate troubleshooting and performance evaluation." (from PER v. 33 no. 3, Dec. 2005 - guest editorial by Dina Papagiannaki and Yin Zhang) ∞ "Legend says that Archimedes remarked, on the discovery of the lever, 'Give me a place to stand and I can move the world.' Today, computing pervades all aspects of society. 'Science' and 'computational science' have become largely synonymous, and computing is the intellectual lever that opens the pathway to discovery in diverse domains. As new discoveries increasingly lie at the interstices of traditional disciplines, computing is also the enabler for scholarship in the arts, humanities, creative practice and public policy. Equally importantly, computing supports our critical infrastructure, from monetary and communication systems to the electric power grid. With such pervasive dependence, computing system reliability and performance are ever more critical. Although the mean time before failure (MTBF) of commodity hardware components (i.e., processors, disks, memories, power supplies and networks) is high, their use in large, mission critical systems can still lead to systemic failures. Our thesis is that the 'two worlds' of software - distributed systems and sequential/parallel systems - must meet, embodying ideas from each, if we are to build resilient systems. This talk surveys some of these challenges and presents possible approaches for resilient design, ranging from intelligent hardware monitoring and adaptation, through low-overhead recovery schemes, statistical sampling and differential scheduling and to alternative models of system software, including evolutionary adaptation." (Daniel A. Reed, "Performance and Reliability: the Ubiquitous Challenge," from PER v. 34, no. 1, Jun. 2006) ∞ "I would like to introduce this issue by telling a story. Sometime back in 1997, I wrote a paper on a new idea for improving the response times of HTTP requests at a Web server. The idea was to schedule the HTTP requests so as to favor requests for small files, in accordance with the well-known scheduling policy Shortest Remaining Processing Time (SRPT). The paper was rejected, for many reasons, but the review that stuck in my mind was the one that said, 'Why is this person writing about scheduling? Scheduling is dead.' According to this reviewer, everything that would ever be known about scheduling was already described in the beautiful Theory of Scheduling book, written in 1967, by Conway, Maxwell, and Miller." (Mor Harchol-Balter in PER v. 34 no. 4, March 2007 Special Issue on Perspectives in Scheduling)
-
- 1990s
-
∞ The Internet became a lot more prominent (and crowded!) with the invention of HTTP and the Web (1989). The ability of home users to access the Internet through Internet Service Providers (circa 1993) made it (theoretically) possible for anyone with a computer and a modem to send and receive e-mail, make Web pages, and access information online. (Complete pandemonium ensued!) ∞ At an international workshop on Performance Evaluation held in Saarbrucken, Germany in September 1997, a group of performance evaluation specialists were invited to provide input on the history and current state of the field. Some of the successes mentioned were improvements in computer architecture, operating systems, and data storage; the development of TCP/IP and the IEEE 802 suite of protocols; and the increased application of performance evaluation theories to end-user software packages. On the other hand, it was noted that performance evaluation methodology was still a seriously neglected part of computer science education, and that often systems were only subjected to evaluation when it was obvious they weren't working properly. Also, the continued division between theory and practice was commented upon. (from the editorial of PER v. 25, no. 4, Mar 1998) ∞ From PER v. 27 no. 3 (Dec. 1999): "The complexity of computer systems, networks and applications, as well as the advancements in computer technology, continue to grow at a rapid pace. Mathematical methods and analysis are playing an important role in research studies to investigate fundamental issues and tradeoffs at the core of real design and performance problems in order to impact the design and implementation of complex computer systems, networks and applications." Historical Notes: ∞ In 1990, as reported in PER v. 17 no. 2 (August 1990), SIGMETRICS had about 2,000 members, and a budget surplus of $90,000 US (equivalent to completely funding 1.5 SIGMETRICS conferences). ∞ In 1990-91, the hot topic appeared to be benchmarking. Three issues of PER had extensive discussions of benchmarking in general, and the UnixWorld/Neal Nelson benchmark method in particular (specifically, whether it actually provided accurate results). The debate was quite heated. ∞ Editor Blaine Gaither indicates that PER is now able to accept submissions electronically, "directly from USENET" (PER v. 18 no. 2, 1990). ∞ In 1992, while SIGMETRICS had a healthy surplus in the bank, ACM as a whole was operating in the red. All the SIGs were requested to maintain a balance in the SIG general fund, which ACM was planning to temporarily use to cover the deficit. Needless to say, this didn't sit too well with a lot of SIG boards, particularly those who had a surplus they weren't allowed to spend. In the editorial pages of PER v. 19 no. 3 (Feb 1992), SIGMETRICS Chair Michael Molloy discussed this and other matters pertaining to the future course of SIGMETRICS: - Whether SIGMETRICS ought to change its name, since it was frequently mistaken for a SIG focused on software metrics rather than performance evaluation
- How involved SIGMETRICS ought to be with performance benchmarking groups (e.g. SPEC, GPC, the Computer Measurement Group, and IFIP Working Group 7.3)
- How SIGMETRICS could reconcile the (persistent) conflict between experimental and theoretical work
- How to get system designers and analysts to use performance evaluation techniques before and during the construction of computer systems, rather than attempting to fix performance problems after the fact
- What SIGMETRICS members would like to see in future issues of PER
- How much SIGMETRICS should charge for membership dues
- How much SIGMETRICS should charge for conference fees
- What members would like to see at future SIGMETRICS conferences
∞ At the SIGMETRICS/Performance 1992 conference in Newport, RI (June 1-5, 1992), SIGMETRICS members discussed the possibility of holding future conferences in Europe, whether to offer cheaper memberships to students, and whether to change the SIG's name to SIGPERF. Evidently, nobody was really interested in a name change. ∞ According to PER v. 20, no. 3, the membership of SIGMETRICS was approximately 1,800 as of March 1993, with most SIGMETRICS members also belonging to other ACM SIGs. The most prominent overlaps were with SIGOPS (41% of SIGMETRICS members also belonged to SIGOPS), SIGARCH (36%), SIGCOMM (30%), SIGSOFT (30%), SIGPLAN (26%), SIGMOD (20%), and SIGSIM (16%). These statistics are from June 1997, when SIGMETRICS had 1364 members. ∞ During the 1990s, membership remained stable, but didn't grow. The PER publication schedule also suffered, with only one or two issues a year (including SIGMETRICS proceedings) being released between 1994 and 1997. ∞ At the SIGMETRICS 1995 conference, held in conjunction with IFIP Performance 1995 from May 15-19 in Ottawa, ON, the organizers introduced a new type of session called "Hot Topics." These sessions were intended to provide a forum for more spontaneous, less formal presentation of a topic selected by a group of authors. ∞ 1996 marked the first year that papers could be submitted for SIGMETRICS electronically (!), as well as the first year that SIGMETRICS was held as part of a large, multi- conference event with the Federated Computing Research Conference (FCRC) in Philadephia, PA, May 23-26. In addition, this was the first year that an award was presented the Best Integration of Systems and Theory, as well as an award for the Best Simulation Paper. The latter award was presented by ACM Transactions on Modeling and Computer Simulation (ToMACS). ∞ On May 1, 1999, the first Workshop on MAthematical performance Modeling and Analysis (MAMA '99) was held in Atlanta, GA, in conjunction with ACM SIGMETRICS '99 and FCRC '99. The workshop was sponsored by ACM SIGMETRICS with support from IBM Research. The workshop organizers sought to bring together researchers working on the mathematical, methodological and theoretical aspects of performance analysis, modeling and optimization.
-
- 1980s
-
∞ During the 1980s, the invention of the personal computer (and the subsequent steady drop in the cost of purchasing one) meant that increasing numbers of people had a computer at home for word processing, keeping track of home finances, playing games, or even communicating with other computers. The release of the Apple MacIntosh, with its user-friendly graphic interface, made personal computers even more popular and less intimidating. ∞ PER v. 9 no. 3 (Fall 1980) printed benchmarking results for a contemporary computer: "Digital Equipment Corporation's Datasystem 355 as configured for our tests consists of a PDP ii/34A central processor, 256K bytes of central memory (128K is standard), 10.4 megabytes of hard-disk storage, an eight-line multiplexer, a VT100 console terminal, and an LA180 line printer. Priced at $43,550 (three additional VT100 terminals would bring the price to $49,700), our benchmark tests revealed the following..." Anecdotes: ∞ "As a retirement present, I bought myself a rather complete micro system so that I can do my editorial work, prepare class outlines and, since I have one large lecture class, I use my micro to produce scrambled tests - helps keep the students honest. And to really have fun I have added a number of software compilers, including three different Basics, Fortran 77, Forth, Pascal, assemblers, etc. Just in case there are some other "bugs" out there with a Z-80 based machine, I have some good notes in GPSS, assorted sorts and teaching simulation models. Although I have worked at this machine for a number of months I still must get used to the different text editors I have. Therefore, excuse my typos - can't blame the computer for that!" (Editor Harold Joseph Highland, PER v. 10 no. 4, Winter 1981-82) Historical Notes: ∞ During the early-to-mid 1980s, sparsely-attended conferences and a drop in membership resulted in a drop in revenues. This led to cutbacks in the length and publication schedule of PER, as well as an increase in SIGMETRICS membership fees from $9 to $12. ∞ In 1985, the SIG Board of ACM voted to change the term of office for SIG officers from one year to two, with a possible two-year extension provided that 2/3 of the SIG's board members agreed. ∞ In 1986-87, SIGMETRICS faced a bit of a cash crunch owing to having to move the SIGMETRICS 1986 conference dates from August to May in order to accommodate IFIP Performance 1986, and consequently having to print two sets of SIGMETRICS proceedings in the same fiscal year. The 1986 fiscal year ended with SIGMETRICS having only $177 in the bank (see PER v. 14 no. 3-4). For the next few years, the PER publication schedule was cut to two or three issues per year instead of four. ∞ SIGMETRICS 1987 (May 11-14) was held in Banff, with the conference banquet at the Banff Springs Hotel. There are some very entertaining photos in PER v. 15 no. 2 (Aug 1987) here.
-
- 1970s
-
∞ Performance evaluation during the 1970s was just as important as it is now, but for different reasons. Back then, only universities and large companies had their own computers, and the things often took up multiple floors of buildings and required days (!) to run a program. A computer system was also a major investment, so being able to run programs faster and more efficiently could save hours of computing time and thousands of dollars. (This was even more crucial for companies or researchers who didn't have access to their own computer and had to rent time on someone else's system.) ∞ The First National SIGME Symposium on Measurement and Evaluation was held February 26-28, 1973 in Palo Alto, California (General Chair: J.D. Madden; Program Chair: T.B. Pinkerton). Over 200 people attend. Tension arises between members whose main interest is in creating models of idealized systems and those who are more interested in evaluating existing systems. This tension never really goes away. Anecdote: ∞ From "On Comparing Apples and Oranges, or, My Machine is Better Than Your Machine", a PER article in September 1972: - "In a recent comparison test, six computer manufacturers were asked to code a particular program loop to run as quickly as possible on their machine. Presumably conclusions about the merits of the machines were to be drawn from the resulting code."
- "We have reduced the number of Instructions for the loop by an average of one instruction per machine, a 15% decrease. It appears that conclusions might more appropriately be drawn about manufacturers' software."
Historical Notes: ∞ SICME became SIGME (Special Interest Group on Measurement and Evaluation) on December 8 1972, following a successful mail-in petition campaign to convert prior to the Committee's first symposium. Apparently, 100 signatures were required for conversion, and over 135 were received before ACM headquarters stopped counting. ∞ At the 1972 FJCC, a decision is made to change the name of SIGME to SIGMETRICS. ∞ The first SIGMETRICS elections are held in the fall of 1973. There are now over 1,000 members, and the organization has a cash surplus of over $3,000 US. ∞ The first two issues of PER in 1974 are delayed due to materials being lost in the post (!). Editor J. C. Browne offers humble apology. ∞ Steady growth through the 1970s and continued tension between modelers and evaluators. There is also a certain degree of territorial conflict with other SIGs, notably SIGSIM, which also has a major focus on modeling. ∞ Publication of PER through the 1970s was irregular, with reasons for the delays ranging from postal (!) difficulties, lack of submissions, and the editor (Harold Joseph Highland) suffering from a heart attack. ∞ Proposed bylaws for SIGMETRICS are first printed in PER v. 8 no. 4 (Winter 1979-80), bringing SIGMETRICS bylaws in line with those of ACM. ∞ Also in this issue, the SIGMETRICS Chair, Gary Nutt, requests submissions from members and points out that the purpose of PER is to showcase works in progress in a timely manner. Despite this, for the next decade, members continue to send in publication quality (and size) articles, resulting in alternating backlogs and shortages of articles.
-
- Origins
-
∞ SIGMETRICS began its existence as SICME (Special Interest Committee on Measurement and Evaluation), which was created by the ACM council on November 19, 1971 at the Fall Joint Computer Conference (FJCC). The original rationale for the SIG was: - "to provide a means for the timely dissemination of relevant information about measurement and evaluation"
- "to provide an arena for discussion between researchers engaged in developing methodologies and users seeking new or improved techniques for the analysis of computer systems"
- "to focus scattered research efforts in a significant area which has not previously been well-represented"
∞ The first SICME newsletter, known as Performance Evaluation Review (PER), was published in March 1972. ∞ SICME had over 500 members by the publication of the second issue of PER. Over 40% of these SIG members were non-ACM members.
-
