The ever growing need to efficiently store, retrieve and analyze massive datasets, originated by very different sources, is currently made more complex by the different requirements posed by users and applications. Such a new level of complexity cannot be handled properly by current data structures for big data problems.

To successfully meet these challenges, we launched a project, funded by the Italian Ministry of University and Research (PRIN no. 2017WR7SHH), that will lay down the theoretical and algorithmic-engineering foundations of a new generation of Multicriteria Data Structures and Algorithms. The multicriteria feature refers to the fact that we wish to seamlessly integrate, via a principled optimization approach, modern compressed data structures with new, revolutionary, data structures learned from the input data by using proper machine-learning tools. The goal of the optimization is to select, among a family of properly designed data structures, the one that “best fits” the multiple constraints imposed by its context of use, thus eventually dominating the multitude of trade-offs currently offered by known solutions, especially in the realm of Big Data applications.

What is a multicriteria data structure?

A multicriteria data structure, for a given problem $P$, is defined by a pair $\langle \mathcal F, \mathcal A \rangle_P$ where $\mathcal F$ is a family of data structures, each one solving $P$ with a proper trade-off in the use of some resources (e.g. time, space, energy), and $\mathcal A$ is an optimisation algorithm that selects in $\mathcal F$ the data structure that best fits an instance of $P$.

Family of data structures
Family of data structures
Computational resources
Computational resources
Optimisation algorithm
Optimisation algorithm

For more details on the project, have a look at its full description here.

Publications

Quickly discover relevant content by filtering publications.
Learned data structures. Oneto L., Navarin N., Sperduti A., Anguita D. (eds) Recent Trends in Learning From Data. Studies in Computational Intelligence, vol 896. Springer, 2020.

PDF DOI

Software

BigRePair

A grammar compressor for huge files with many repetitions.

Block-epsilon tree

A compressed rank/select dictionary exploiting approximate linearity and repetitiveness.

CORENup

A Combination of Convolutional and Recurrent Deep Neural Networks for Nucleosome Positioning Identification.

FADE

An extensible framework to efficiently compute alignment-free functions on a set of large genomic sequences.

FASTdoopC

A general software framework for the efficient acquisition of FASTA/Q genomic files in a MapReduce environment.

FastKmer

A SPARK software system for the collection of $k$-mer statistics.

LA-vector

A compressed bitvector/container supporting efficient random access and rank queries.

Learned sorted table search

A software library to speed-up sorted table search procedures via learning from data.

PFP Data Structures

Data structures supporting Longest Common Extensions and Suffix Array queries, built on the prefix-free parsing of the text.

PGM-index

A data structure enabling fast searches in arrays of billions of items using orders of magnitude less space than traditional indexes.

PyGM

Python library of sorted containers with state-of-the-art query performance and compressed memory usage.

sHAM

Compression strategies and space-conscious representations for deep neural networks.

Invited talks

Learned Indexes
A rigorous approach to design learned data structures
On the compressed indexing of dictionaries of keys for massive key-value stores
A tutorial on learning-based compressed data structures
Theory and practice of learning-based compressed data structures
Learned and Compressed Data Structures, challenges on Storage Systems
Learned and Compressed Data Structures
The future of data structures: data-aware and self-designing
Life sciences and algorithmic design: speed and accuracy in small space
Hybrid Data Structures and beyond
The evolution of searching data structures

Events

 
 
 
 
 

The future of compressed data structures, 20 years after the FM-index

July 19, 2022 – July 20, 2022 Lipari, Italy
Organised by Paolo Ferragina and Giovanni Manzini. Info
 
 
 
 
 

2020 Student Challenge @ Algorithm Engineering UniPi course

November 4, 2020 – February 10, 2021 Pisa, Italy
Organised by Paolo Ferragina and Giorgio Vinciguerra. Info

Meetings & Reports

 
 
 
 
 

Project end

August 28, 2022 (postponed to February 28, 2023 because of COVID-19)
 
 
 
 
 

Report of the second year

September 1, 2021
 
 
 
 
 

Report of the first year

September 1, 2020
 
 
 
 
 

Kickoff meeting

October 14, 2019 Video conference

Minutes of the meeting

 
 
 
 
 

Project start

September 1, 2019

People

Principal Investigators

Avatar

Marco Frasca

Assistant professor

Avatar

Raffaele Giancarlo

Full professor

Università di Pisa

Avatar

Davide Bacciu

Assistant professor

Avatar

Andrea Guerra

PhD Student

Università degli Studi di Milano

Avatar

Marco Frasca

Assistant professor

Avatar

Dario Malchiodi

Associate professor

Avatar

Marco Mesiti

Associate professor

Avatar

Paolo Perlasca

Assistant professor

Università degli Studi di Palermo

Avatar

Raffaele Giancarlo

Full professor

Avatar

Giosuè Lo Bosco

Associate professor

Avatar

Simona E. Rombo

Associate professor

Università degli Studi del Piemonte Orientale “Amedeo Avogadro”

Avatar

Lavinia Egidi

Associate professor

Avatar

Manuel Striani

Postdoc research fellow