LAWA · Longitudinal Analytics of Web Archive data

To understand what is required to support new innovative Internet applications, a solid understanding of Internet content characteristics (size, distribution, form, structure, evolution, dynamic) is necessary. The LAWA project (LAWA - Longitudinal Analytics of Web Archive data) will build an Internet-based experimental testbed for large-scale data analytics. Its emphasis is on developing a sustainable infrastructure, scalable methods, and easily usable software tools for aggregating, querying, and analyzing heterogeneous data at Internet scale. For decades, compute power and storage have become steadily cheaper, while network speeds, although increasing, has not kept up. The result is that data is becoming increasingly local and thus distributed in nature. It has become necessary to move more analysis to the data, not the reverse. The Internet is already, a long-scaled heterogeneous complex system.LAWA will federate distributed FIRE facilities with the rich centralized Web repository of the European Archive, to create a Virtual Web Observatory and use Web data analytics as a use case study to validate our design. The outcome of our work will enable Web-scale analysis of data, to unlock large-scale study of the content aspect of the Internet and bring this dimension on the roadmap of Future Internet Research. In four workpackages we will extend the open-source Hadoop parallel query management software by novel methods for data access and import, develop new methods of distributed storage with indexing, offer scalable aggregation, mine metadata and text along the time dimension, and advance the art of automatic classification of Web contents.LAWA adds value to the FIRE community by offering access to very large datasets across thousands of storage and processing nodes, with advanced methods and open-source tools for intelligently analysis at Internet scale enabling research for the Future Internet to take into account the challenge of content explosion.