emeryberger/csrankings

A web app for ranking computer science departments according to their research output in selective venues, and for finding active faculty across a wide range of areas.

Emery Berger

Professor, Computer Science, Univ. of Massachusetts Amherst. My students, collaborators, and I in the PLASMA lab build innovative and impactful systems, especially focused on performance, reliability, and security. Among other things, I am also the creator/maintainer of https://CSrankings.org. For the academic year 2023/2024, I am on sabbatical and am an Amazon Scholar at Amazon Web Services.

Recent blog posts

• Coping with CoPilot (SIGARCH/SIGPLAN)
• Drop Whatever You're Researching and Start Working on Crypto! (SIGARCH)
• A Checklist Manifesto for Empirical Evaluation (SIGARCH/SIGPLAN)
  (SIGPLAN Empirical Evaluation Checklist PDF)

Links

• PLASMA lab GitHub:
• Personal GitHub:
• Mastodon:
• Twitter: @emeryberger
• YouTube:
• emeryberger.com, Medium blog
• Publications: Google Scholar, DBLP

Selected software from me and my collaborators (for a full list, see PLASMA)

• Scalene: A state-of-the-art CPU+GPU+memory profiler for Python, with AI-powered optimization suggestions
  
• Coz: A causal profiler that tells you where to optimize your code (C/C++/Rust/Swift/Java)
  (just counting Rust crate downloads)
• CSrankings: Ranks the world's CS departments
  
• Slipcover: Nearly zero-overhead code coverage analysis for Python
  
• ChatDBG: AI-assisted debugging in lldb, gdb, and pdb. Uses AI to answer 'why'
  
• Commentator: Automatically comments Python code, adding docstrings and type annotations
  
• CWhy: ("See why") Explains and suggests fixes for C/C++/Rust compiler error messages.
  
• SQLwrite: SQLite3 with natural language to SQL, recommends indexes to optimize queries
  

Selected Talks

Performance Matters (Strange Loop 2019)
The second most popular Strange Loop video of all time!

Python Performance Matters (Strange Loop 2022)

Scalene (PyCon 2021)

How to Get Your Research Adopted (PLDI keynote)

Selected Papers

• Triangulating Python Performance with Scalene (OSDI 2023, Best Paper Award)
  Introduces Scalene, a profiler that highlights Python performance issues. [software]
• Coz: Finding Code that Counts with Causal Profiling (Best Paper Award SOSP 2015, CACM Research Highlight)
  Introduces causal profiling and the Coz profiler. [software]
• Mesh: Compacting Memory Management for C/C++ Applications (PLDI 2019)
  A drop-in replacement for malloc that eliminates fragmentation. [software]
• BLeak: Automatically Debugging Memory Leaks in Web Applications (PLDI 2018, CACM Research Highlight)
  Detects memory leaks in web applications automatically. [software] incorporated into Android Studio
• Stabilizer: Statistically Sound Performance Evaluation (ASPLOS 2013)
  Randomizes everything to see if performance changes are real. incorporated into WebAssembly benchmarking infrastructure
• Reconsidering Custom Memory Allocation (OOPSLA 2012, Most Influential Paper)
  Shows which kinds of custom memory allocators are useful for improving performance (TL;DR - not many).
• Hoard: A Scalable Memory Allocator for Multithreaded Applications (ASPLOS 2000, ASPLOS Most Influential Paper)
  The first scalable malloc. [software] incorporated into the Mac OS X allocator

Bio

Emery Berger is a Professor in the Manning College of Information and Computer Sciences at the University of Massachusetts Amherst, the flagship campus of the UMass system. He graduated with a Ph.D. in Computer Science from the University of Texas at Austin in 2002. Professor Berger has been a Visiting Scientist at Microsoft Research and at the Universitat Politècnica de Catalunya (UPC) / Barcelona Supercomputing Center (BSC).

Professor Berger’s research spans programming languages, runtime systems, and operating systems, with a particular focus on systems that transparently improve reliability, security, and performance. He and his collaborators have created a number of influential software systems including Hoard, a fast and scalable memory manager that accelerates multithreaded applications (used by companies including British Telecom, Cisco, Crédit Suisse, Reuters, Royal Bank of Canada, SAP, and Tata, and on which the Mac OS X memory manager is based); DieHard, an error-avoiding memory manager that directly influenced the design of the Windows 7 Fault-Tolerant Heap; and DieHarder, a secure memory manager that was an inspiration for hardening changes made to the Windows 8 heap (see this map of the landscape of memory management research for an overview). He also created and maintains the widely-used CSrankings website.

His honors include a Microsoft Research Fellowship, an NSF CAREER Award, a Lilly Teaching Fellowship, the Distinguished Artifact Award for PLDI 2014, the Most Influential Paper Award at OOPSLA 2012, the Most Influential Paper Award at PLDI 2016, the ASPLOS 2019 Influential Paper Award, five SIGPLAN Research Highlights, five CACM Research Highlights (1, 2, 3, 4, 5), a Google Research Award, a Microsoft SEIF Award, and Best Paper Awards at FAST, OOPSLA, and SOSP. Professor Berger served two terms (1, 2) as an elected member of the SIGPLAN Executive Committee; he served for a decade (2007-2017) as Associate Editor of the ACM Transactions on Programming Languages and Systems, was Program Chair for PLDI 2016, and served as co-Program Chair of ASPLOS 2021. He was named an ACM Fellow in 2019.

csrankings's Issues

ICS aliasing issue

http://dblp.uni-trier.de/db/conf/innovations/index.html is being treated as http://dblp.uni-trier.de/db/conf/ics/index.html
and is being counted under HPC.

Missing papers

Recent commits (not sure which one specifically, but within the last couple of days) seem to cause some inconsistencies

E.g., http://dblp.uni-trier.de/pers/hd/s/Srisa=an:Witawas has 1 (full) ICSE, but his name no longer appears in the UNL Software Engineer ranking.
Also, http://dblp.uni-trier.de/pers/hd/n/Nguyen:ThanhVu has 3 (full) ICSE papers, but only 2 were counted.

These data were all correct previously.

Empty names and publication count on first result

Hi,
when selecting "Computer architecture ASPLOS, ISCA, MICRO" and deselecting every other option, the EPFL entry is empty, but is still ranked first. It seems there is a display problem.

Screenshot illustrating the situation attached. Browser used: chromium or firefox, with a fresh profile

add one-off joint NLP conference names

Oops I meant to do this as a pull request in order to document it, but I accidentally committed to gh-pages here:

cc7171a

This was found by looking at counts of all <booktitle> lines in dblp.xml and searching for nlp and acl.

These are from one-off instances of when one of the main conferences was held jointly with another conference. The joint names are part of the official titles. Colloquially, researchers at these conferences seem to think of these joint occurrences as instances of the main conference. I totally forgot the 2012 EMNLP was officially joint with CoNLL, for example.

I checked https://www.aclweb.org/aclwiki/index.php?title=Conference_acceptance_rates and confirmed acceptance rates are about the same (I think in these instances reviewing practices proceed in the same way the main conference does them).

Also I removed the "SIGDAT" comment since it isn't an ACM thing like all the other SIG* instances in this file. SIGDAT is a group within ACL (that organizes EMNLP last I checked). ACL, ICML (organization: IMLS), NIPS, and WWW (organization) aren't associated with ACM as far as I know...

add more conferences + checkboxes for each

This would keep the current system of top conferences for each area, but would let people drill down further into subareas. I see this as being revealed by another clickable triangle below each area, though handling this elegantly in the UI is not obvious.

Add Design Automation to the list of CS areas

Rationale

SIGDA is part of ACM. Many of the "top 20" schools in the US have faculty in CS that have faculty publishing in design automation conferences.

Thanks,
Simha

Evolutionary Computation area?

Consider including Evolutionary computing as an AI sub-field. One colleague suggested the journal "Evolutionary Computation" as the leading venue; the suggested candidates for the next ones were either the journals "IEEE Transactions on Evolutionary Computation" and "Genetic Programming and Evolvable Machines, or the CEC and GECCO conferences. Need to consult with others in the field to dial this in.

Unjust Geometric Mean Formula

The geometric mean formula automatically assigns the adjusted count of an area to 1 if it was zero previously.

This seems unfair to universities which have non-zero very small count in that area. It would just have been better to not have published at all.

@emeryberger

Add collaboration graphs

With the existing data, it should now be straightforward to generate collaboration graphs for each institution. Cross-institution would also be doable - in fact, a whole map of collaboration networks across CS would in theory be easy to do. (H/T @dbarowy).

(Here's one from a long time ago as an example - https://people.cs.umass.edu/~emery/misc/collaboration/).

Any chance of a "CS Education" Area?

This could represent SIGCSE, TOCE, CSE, ITiCSE, ICER, etc. I know it's not the most popular subfield, but we're growing in importance and popularity.

special-case handling for ICSE

Papers past a certain number are not research contributions; these should be filtered out (data contributed by Zhendong Su).

conference year last page number for research papers
ICSE
2016 n/a
2015 n/a --- but only ICSE (1) contains research papers
2014 n/a
2013 851
2012 957
2011 620
2010 544
2009 550
2008 n/a
2007 510
2006 411
2005 478
2004 n/a
2003 477
2002 534
2001 502
2000 518
1999 582
1998 419
1997 535
1996 mixed tracks but the only one that isn't ruled out by paper length is Basili's keynote paper, which has 8 pages
1995 mixed tracks Michael Jackson's keynote paper has 10 pages
1994 mixed tracks a most influential paper and a keynote paper (both have >= 6 pages)
1993 n/a
1992 360 three nvited papers before page 38, where the first research paper starts

revisit inclusion criteria for non-CS faculty

How and should we include faculty at institutions who are not in a CS / joint EECS or similar academic unit (currently, they are not included).

What would be the procedure to add a faculty?

Hi,
pretty much everything is in the title: my faculty is currently missing and I wonder what would be the necessary steps to add it to the website. List of researchers ? Email adresses ?

Thank you.

Add SIGGRAPH Asia conference

Please add the SIGGRAPH Asia (SA) conference to the Computer Graphics category. The SA conference utilizes the exact same reviewing process as SIGGRAPH and has the same quality level. Both conferences have their complete proceedings published as issues of ACM Transaction on Graphics.

suggest to add ICDM and SDM to machine learning and data mining

False positive entity merge on DBLP: "Daniel Sheldon"

http://dblp.uni-trier.de/pers/hd/s/Sheldon:Daniel

This appears to be at least two different people.

I feel ridiculous filing this as an issue, but I guess it's test cases for a research project (or some sort of modification of the database...)

faculty with two affiliations

How should faculty with two affiliations be handled?

Commit 8e710a63 removed duplicated affiliations of some faculty. Some of them, however, are affiliated with two institutions, spending time in both and specifying them as their affiliations in papers:

• Ran Canetti (e.g., https://eprint.iacr.org/2014/243.pdf)
• Kyle Jamieson (e.g., http://dl.acm.org/citation.cfm?id=2906417)
• Nassir Navab (e.g., https://www.ncbi.nlm.nih.gov/pubmed/26984551)
• Ronitt Rubinfeld (e.g., http://www.jmlr.org/proceedings/papers/v49/aliakbarpour16.pdf)
• Nir Shavit (e.g., http://dl.acm.org/citation.cfm?id=2935825)

Wrong number of raw publications

This is for K. Srinathan of IIIT Hyderabad. The pie chart shows a different value from the raw number of publication.

pie chart discrepancy vs. table

1. Select all AI areas
2. go to UMass
3. pie chart for "Brendan T O'Connor" says 21 but the raw count says 7

double escaping on google search for person name

after clicking on a person's name (for people where it tries to launch a google search)

i think this is a double escaping issue -- the %20 things appear in the query box ui

suggest to add ICDE to Databases

CSrankings is missing half my papers (on DBLP)

(6/20: updated the numbers)

I don't care personally and feel like it's not even worth addressing such a drop in the bucket and it may not affect rankings much, but anyway...

The page on DBLP that corresponds to me is:
http://dblp.uni-trier.de/pers/hd/o/O=Connor:Brendan_T=

For papers fitting into the NLP categories (ACL, NAACL, EMNLP), I think it is completely correct.
It has 8 papers total. But the CSRankings interface says I have 6. I'm not sure how to see which exactly are missing?

Here's the list of papers from DBLP. I figure the issue is alternate naming of pub venues? There's some annoying history and conflicting naming schemes for some of the conferences (e.g. "HLT-NAACL" is now often just called "NAACL") so you end up seeing citations on internet or gscholar or i'm sure dblp where the same conference is given pretty different names by different people. bleah

Khanh Nguyen, Brendan O'Connor:
Posterior calibration and exploratory analysis for natural language processing models. EMNLP 2015: 1587-1598

David Bamman, Brendan O'Connor, Noah A. Smith:
Learning Latent Personas of Film Characters. ACL (1) 2013: 352-361

Brendan O'Connor, Brandon M. Stewart, Noah A. Smith:
Learning to Extract International Relations from Political Context. ACL (1) 2013: 1094-1104

Olutobi Owoputi, Brendan O'Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, Noah A. Smith:
Improved Part-of-Speech Tagging for Online Conversational Text with Word Clusters. HLT-NAACL 2013: 380-390

Kevin Gimpel, Nathan Schneider, Brendan O'Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, Noah A. Smith:
Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments. ACL (Short Papers) 2011: 42-47

Dani Yogatama, Michael Heilman, Brendan O'Connor, Chris Dyer, Bryan R. Routledge, Noah A. Smith:
Predicting a Scientific Community's Response to an Article. EMNLP 2011: 594-604

Jacob Eisenstein, Brendan O'Connor, Noah A. Smith, Eric P. Xing:
A Latent Variable Model for Geographic Lexical Variation. EMNLP 2010: 1277-1287

Rion Snow, Brendan O'Connor, Daniel Jurafsky, Andrew Y. Ng:
Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. EMNLP 2008: 254-263

More statistics and citations

Show standard deviation, optionally including folks in the department that did not contribute (i.e., a dead meat meter). Maybe add some other funky statistics and data like citations as well. Adding citations would involve downloading papers and parsing citations (this might be hard and involve quite some scripting).

People sort

Rank people and not just departments (this would allow us to compare people across departments).

Adding additional institutions

The coverage of Israel seem to be quite lacking, with only Tel Aviv University and the Technion being represented. What is the procedure of adding other institutions (Bar Ilan University, Hebrew University, Ben Gurion University)?

Make affiliations relevant per year

It seems this system does not account for arrivals/departures of faculty? If so, then rankings computed up to any year in the past (e.g., from 2000-2010) would be flawed because they would be computed using the present roster, not the past roster (i.e., the one in 2010). Generally speaking, one might think that the ranking from years XXX-YYY should computed from the average pub count of the actual faculty present during each of those years.

Include CSCW

I note that CSCW is not included in the HCI rankings.

If the premier conference on collaborative and social computing is excluded, then HCI researchers focusing on that area can only help their university's rankings with that work that they decide to send to CHI instead of the more focused CSCW conference. This seems like a perverse incentive.

I would recommend either re-evaluating the 3-conference rule or splitting HCI into subfields. It is a sufficiently broad topic that capturing it in 3 conferences does not seem realistic.

Not all faculty of a university is shown

Ashish Choudhury from IIIT Bangalore has one publication in CRYPTO but is not shown in the UI.
He also is present in faculty-affiliations.csv.

This bug arose after the last bug fix.

Adding Top 3 journals as well

Hi

I was just wondering why we have only included the Top 3 conferences and not included the top journals in respective fields. Could you please explain your thinking?

Thanks and regards
Dhruv

More strict ordering in "Average count"

As of now, geometric mean is shown and compared upto one decimal place which leads to a lot of universities clustering together.

I propose that the number of decimal places upto which it is compared can be increased, say 8 decimal places. Although, the number of digits shown can be kept the same for keeping the UI clean.

@emeryberger

NDSS missing from Security section

I think NDSS should be one of the Computer Security venues, and looking at util/build-citations.py it is. However in util/csrankings.py it is not present (instead USENIX Security is there twice). NDSS is also not listed in index.html.

Systematic way to find areas/sub-areas?

Is there some way to at least semi-systematically identify key sub-areas/publications that aren't currently included but should be? There are, for example, lots of systems sub-areas and publication venues listed, but few in other areas.

One way would be to come at it from known top researchers. How hard would it be to check the publication score of all of the ACM fellows elected in the last decade, for example?

consider adding another AI venue

There are currently two, so there is room for one more. Proposed candidates include AAMAS, ICAPS, KR, UAI, SOCS, CP.

revisit third conference for SE

inclusion of ASE is controversial; arguments made by some that is not up to the level of ICSE/FSE

permalinks

Are there permalinks for particular selection settings? Would be a nice-to-have -but-not-essential feature.

Number of faculty is incorrectly displayed on the site

This snapshot shows that IIIT Bangalore has 6 members listed but just 5 is shown as the number.

Asian universities

Do you plan to include universities in Asia as well? I can think of several, especially in AI/NLP/ML that contribute significantly to top conferences.

Suggestion to add WSDM to "web" area

WSDM is one of the 3 most selective web search and IR conferences
http://dblp.uni-trier.de/db/conf/wsdm/

Addition of Asia

Hi,

There are quite a few universities in Asia (even in the current CS rankings list). List below:

1. Technion - Israel Institute of Technology
2. Tel Aviv University
3. Hebrew University of Jerusalem
4. IISc Bangalore

Can you please create a separate entry for asia and list these universities under it?

Thanks,
Murali Krishna

google search links not working

Hi

the google search links of professors, (for example, **_

https://www.google.co.in/search?q=Eileen%2520Kraemer%2520Clemson%2520University&btnI=&gws_rd=cr&ei=xrkxWLbDHcHWvASFr73gCw

_** )

do not work. there is an extra "%25" being embedded between spaces.

check whether Multimedia meets inclusion criteria

conferences: ACM MM, possibly MMSys

Add FAST to OS?

hover display shouldn't fade

the hover view of individual researchers is great!

it would be even better if it didn't disappear after N seconds. several times we found ourselves in the middle of reading one then it disappeared.

Suggestion to add RecSys in "data mining" area

One of the top conferences in data mining & analysis.

https://scholar.google.com/citations?view_op=top_venues&hl=el&vq=eng_datamininganalysis&venue=4-w_STT7RmEJ.2016

Many VIS publications not in database

This might be just an update issue, but a lot of VIS papers are not (yet) in the csrankings database:

None of the Utah VIS faculty appear in this screenshot as the papers seem to be missing.

Computational Biology/Bioinformatics area

The top venues are places like Science, Nature, and PNAS. Those are all very broad and not indexed by dblp, but are from what I understand where the best work is published. I'm not sure how accurate using lower-impact but more focused publications would be, but it is an area many CS people work in.

consider adding SIGBED (embedded systems) area and conferences

From a mail:

[...] ACM SIGBED (Special Interest Group on Embedded Systems), http://sigbed.blogspot.com, [...] is a very vibrant group, encompassing embedded systems, real-time systems, cyber-physical systems.
Since csrankings.org does not list Embedded Systems, I would like to ask how it can be added. There are a few premier conferences that can be listed under embedded systems/real-time/cyber-physical systems, if the Embedded Systems area is added under Systems.

The top three conferences I would like to suggest are as follows:

1. RTSS (IEEE Real-Time Systems Symposium)
2. RTAS (IEEE Real Time and Embedded Technology and Applications Symposium) — name changed from Real-Time Technology and Applications Symposium
3. EMSOFT (ACM International Conference on Embedded Software) - name changed from Workshop to Conference in 2003 - Top embedded software conference, flagship conference of ACM SIGBED

most of my papers on DBLP are missed by csrankings

http://dblp.uni-trier.de/pers/hd/w/Wang_0010:Wei

Add "North America" as a region

Have a new region for North America that bundles together USA and Canada.

Consider a third HPC venue

Candidates for a third are HPDC, SPAA, and probably ICS. There may not be an easy pick between these, so staying with just the current two (PPoPP and SC) may be the best option.