Netflix Prize and Privacy

Netflix has released a sample of data publicly and has offered a significant prize to predict peoples' rating for movies better than their baseline solution. The data comprises customers and their ratings of movies and the times the ratings were posted. Customers are anonymized, by removal of immediately identifying information.  Some noise is introduced by some perturbations of the data, and a claim that only a small amount of information from any one customer is included.

In a recent study, the authors experimentally evaluate the following: for each customer c and k of their movie rating and times given, determine the number N(c,k) of other customers who posted "nearly" same ratings at "nearly" same times for these movies? The argument is that for small k, if N(c,k) were small, then customer c can be disambiguated with some effort or knowledge.

The paper appears to be in a preliminary form. From a theoretical point of view, there are nice problems here. Can we propose a probabilistic model for customer ratings and derive an estimate for N(c,k)?

What blogs reveal?

What blogs reveal about one is a deeply technical or deeply philosophical question. They have attracted a lot of interest academically. In computer science, people have abstracted problems based on text attributes, analogies with epidemiology on the spread of "memes" through blogs, studied graph properties and so on. In doing such study, one becomes quite familiar with the literature from people in CS and related fields, but also start to learn how they are approached by other fields such as sociology and psychology. An example is a study on gender, identity and language in teenage blogs, done a couple of years ago. A long (exhaustive?) list of blog related research from all discplines appears on the homepage of a blogging researcher / researching blogger.

Privacy and Confidentiality

Stephen Fienberg gave a nice talk with a statistical perspective on data privacy at the kickoff meeting of DyDAn. He introduced the need for privacy awareness in data handling, and some of the hazards if it is ignored (slides are here). He spoke in detail about giving bounds on cells of a contingency table given certain marginals; this was based on his PNAS paper. He also publicized a upcoming WWW 2007 paper by Backstrom, Dwork and Kleinberg on theoretical analysis of privacy in social networks or lack thereof. Both these papers raise a lot of fascinating questions, and lead to a lot of scope for further study, in statistics and theoretical computer science.

DyDAn Official Launch

DyDAn kicks off officially on Fed 26th with a series of talks. There will be talks by Director of Research Futures in DHS, Deputy Director of Inst for Sc. Computing Research at LLNL, and Stephen Fienberg from CMU. Privacy in data analysis is obviously an important theme. In database research community, there is growing emphasis on this. See the 8 papers in 2 revelant sessions at ICDE 2007 and more are forthcoming in SIGMOD and PODS 2007.

RVACs

One of the key aspects of data analysis is visualization. DHS in collaboration with the National Visualization and Analytics Center (NVAC) has established Regional Visualization and Analytics Centers (RVACs) to provide both research expertise and training and education programs. The research program of the RVACs sounds very exciting, in particular, since they include careful study of collaborative decision making methods.

Other UACs

Besides DyDAn, IDS has announced University Affiliate Centers at:

• University of Illinois
• University of Pittsburgh
• University of Southern California

These UACs explore research topics from multi-modal information synthesis to knowledge integration and methods for summarization of opinions in text. Together with DyDAn, the UACs represent a wide range of researchers, from algorithms and discrete math to knowledge discovery and mining, to statistics and multi-modal informatics. This represents an exciting opportunity for learning new problems, developing new techniques and applying ideas across multiple areas.

One of the focuses of UACs is to train researchers at all levels. So look forward to workshops, REU programs, summer schools and others.

Introducing DyDAn

The Department of Homeland Security DHS) has different ways to support research. In particular, DHS now supports research via the Institute of Discrete Sciences (IDS), a joint project between DHS and Lawrence Livermore National Labs. The Institute has announced four University Affiliate Centers (UACs) for research. DyDAn is the UAC based at Rutgers University in NJ. The focus at DyDAn is on research in Algorithms, Discrete Math and Data Analysis, and applications thereof.

This blog will be a forum for researchers not only at DyDAn, but also other UACs and researchers everywhere to interact with each other. Therefore, this blog will not only feature announcements, but also discuss research problems and progress at DYDAn, other UACs and the general research community on topics of common interest to IDS. We welcome comments, suggestions and discussions, directly through blog comments or via email.