Software Heritage priorities in the 2023 technical roadmap

Starting from 2021, Software Heritage publishes a technical roadmap that is updated yearly. As is customary for many Open Source projects, this roadmap serves both as a guideline for the development team, and as a means to share with the broader user and contributor community the current priorities.

We are investing more effort into this latter aspect of the roadmap: after the blogpost that unveiled the 2023 adition, a dedicated session was dedicated to it during the last ambassadors plenary meeting, when Benoît Chauvet, software engineering manager at Software Heritage, explained the “why” and the “how” of the 2023 technical roadmap.

A key message from that presentation is that the Software Heritage archive keeps growing both in terms of volume and visibility, and this leads to three main priorities: prepare the infrastructure for growth, increase the resilience of the achive and roll out new features and services.

The 2023 technical roadmap milestones materialize these priorities, as follows.

Prepare the infrastructure for growth

• Standardise the identifier schema
  • Publicicly available standard for SWHID v1
• Improve robustness
  • Minimize archival lag
  • Support SHA1 collision issues
  • Extend the scrubbing tools
  • Improve swh-storage pagination
  • Improve support for malformed git commits
  • Create a cost calculator in the vault
• Improve scalability
  • Setup a dynamic infrastructure
  • Setup a continuous deployment infrastucture
  • Evaluate the storage of indexed metadata in a triple-store
• Replacement of key infrastructure components
  • Migrate object storage from ZFS to Ceph
  • Migrate the graph storage from PostgreSQL to Cassandra
  • Design and test a next-generation CI
• Increase processes automation
  • Tooling for takedown notices
  • Automate Add Forge Now
  • Common workflow management tool
  • Graph Export and graph compression in production
  • Webhook-based notifications for long-running user tasks

Increase resilience and coverage

• Develop the mirror network
  • Release the first SWH Mirror at ENEA
  • Develop SWH Mirrors at GRNET and Duisburg-Essen university
• Develop disaster recovery solutions
  • Archive cold copy at CINES in Vitam
• Extend archive coverage
  • Release new Listers/Loader to support more forges and vcs technologies
  • Collect and index forge metadata: target 100% coverage

New features and services

• Presentation
  • Design presentation of metadata in web UI
  • Propose web UI sections for partner collections
  • Self-Host Software Stories software stack
• Traceability
  • Release a first version of the swh-scanner
  • Provenance in production

Doing all this is not an easy task: we are deeply grateful to the sponsors that support our work that provide us the key resources, to the many contributors that help with the technical development, to the ambassadors that help build awareness of the mission, and the broader community.

You too can contribute to the mission of Software Heritage: by becoming a sponsor, tackling scientific challenges through participation in research activities, contributing code to the open source infrastructure the archive, spreading the word by becoming an ambassador, or even by a simple donation.

Whether you’re an individual, a company, or an institution, your support can make a difference in preserving and providing access to humankind’s software heritage. Join us in this vital endeavour!