Software repository

A software repository, or “repo” for short, is a storage location for software packages. Often a table of contents is stored, as well as metadata. Repositories group packages. Sometimes the grouping is for a programming language, such as CPAN for the Perl programming language, sometimes for an entire operating system, sometimes the license of the contents is the criteria.

At client side, a package manager helps installing from and updating the repositories.

At server side, a software repository is typically managed by source control or repository managers. Some of the repository managers allow to aggregate other repository location into one URL and provide a caching proxy. When doing continuous builds many artifacts are produced and often centrally stored, so automatically deleting the ones which are not released is important.

Overview

Many software publishers and other organizations maintain servers on the Internet for this purpose, either free of charge or for a subscription fee. Repositories may be solely for particular programs, such as CPAN for the Perl programming language, or for an entire operating system. Operators of such repositories typically provide a package management system, tools intended to search for, install and otherwise manipulate software packages from the repositories. For example, many Linux distributions use Advanced Packaging Tool (APT), commonly found in Debian based distributions, or yum found in Red Hat based distributions. There are also multiple independent package management systems, such as pacman, used in Arch Linux and equo, found in Sabayon Linux.

As software repositories are designed to include useful packages, major repositories are designed to be malware free. If a computer is configured to use a digitally signed repository from a reputable vendor, and is coupled with an appropriate permissions system, this significantly reduces the threat of malware to these systems. As a side effect, many systems that have these capabilities do not require anti-malware software such as anti-virus software.[1]

Most major Linux distributions have many repositories around the world that mirror the main repository.

Package management system vs. package development process

A package management system is different from a package development process.

A typical use of a package management system is to facilitate the integration of code from possibly different sources into a coherent stand-alone operating unit. Thus, a package management system might be used to produce a distribution of Linux, possibly a distribution tailored to a specific restricted application.

A package development process, by contrast, is used to manage the co-development of code and documentation of a collection of functions or routines with a common theme, producing thereby a package of software functions that typically will not be complete and usable by themselves. A good package development process will help users conform to good documentation and coding practices, integrating some level of unit testing. The table below provides examples of package development processes.

Selected repositories

The following table lists a few languages with repositories for contributed software. The "Autochecks" column describes the routine checks done.

Very few people have the ability to test their software under multiple operating-systems with different versions of the core code and with other contributed packages they may use. For R, the Comprehensive R Archive Network (CRAN) runs tests routinely. To see how this is valuable, suppose Sally contributes a package A. Sally only runs the current version of the software under one version of Microsoft Windows, and has only tested it in that environment. At more or less regular intervals, CRAN tests Sally's contribution under a dozen combinations of operating systems and versions of the core R language software. If one of them generates an error, she gets that error message. With luck, that error message may suffice to allow her to fix the error, even if she cannot replicate it with the hardware and software she has. Next, suppose John contributes to the repository a package B that uses a package A. Package B passes all the tests and is made available to users. Later, Sally submits an improved version of A, which unfortunately, breaks B. The autochecks make it possible to provide information to John so he can fix the problem.

This example exposes both a strength and a weakness in the R contributed-package system: CRAN supports this kind of automated testing of contributed packages, but packages contributed to CRAN need not specify the versions of other contributed packages that they use. Procedures for requesting specific versions of packages exist, but contributors might not use those procedures.

Beyond this, a repository such as CRAN running regular checks of contributed packages actually provides an extensive if ad hoc test suite for development versions of the core language. If Sally (in the example above) gets an error message she does not understand or thinks is inappropriate, especially from a development version of the language, she can (and often does with R) ask the core development-team for the language for help. In this way, the repository can contribute to improving the quality of the core language software.

Language / purpose Package Development Process Repository Install methods Collaborative development platform Autochecks
Haskell Common Architecture for Building Applications and Libraries[2] Hackage
Java Maven[3]
Julia[4]
Common Lisp Quicklisp[5]
.NET NuGet NuGet[6]
Node.js NPM[7]
Perl CPAN PPM[8]
PHP PEAR, Composer PECL, Packagist
Python Setuptools PyPI pip, EasyInstall, PyPM, Anaconda
R R CMD check process[9][10] CRAN[11] install.packages[12] R-Forge[13] Roughly weekly on 12 platforms or combinations of different versions of R (devel, prerel, patched, release) with up to 7 different operating systems (different versions of Linux, Windows, and Mac).
Ruby RubyGems Ruby Application Archive RubyForge
Rust Cargo[14] Crates[15] Cargo[14]
TeX, LaTeX CTAN

(Parts of this table were copied from a "List of Top Repositories by Programming Language" on Stack Overflow[16])

Many other programming languages, among them C, C++, and Fortran, do not possess a central software repository with universal scope. Notable repositories with limited scope include:

  • Netlib, mainly mathematical routines for Fortran and C, historically one of the first open software repositories;
  • Boost, a strictly curated collection of high-quality libraries for C++; some code developed in Boost later became part of the C++ standard library.

Package managers

Package managers help manage repositories and the distribution of them. If a repository is updated, a package manager will typically allow the user to update that repository through the package manager. They also help with managing things such as dependencies between other software repositories. Some examples of Package Managers include:

Popular Package Managers
Package Manager Description
NPM A package manager for Node.js[17]
pip A package installer for Python[18]
APT For managing Debian Packages[19]
Homebrew A package installer for MacOS that allows you to install packages Apple didn't[20]

Repository managers

Relationship to continuous integration

As part of the development lifecycle, source code is continuously being built into binary artifacts using continuous integration. This may interact with a binary repository manager much like a developer would by getting artifacts from the repositories and pushing builds there. Tight integration with CI servers enables the storage of important metadata such as:

  • Which user triggered the build (whether manually or by committing to revision control)
  • Which modules were built
  • Which sources were used (commit id, revision, branch)
  • Dependencies used
  • Environment variables
  • Packages installed

Artifacts and packages

Artifacts and packages inherently mean different things. Artifacts are simply an output or collection of files (ex. JAR, WAR, DLLS, RPM etc.) and one of those files may contain metadata (e.g. POM file). Whereas packages are a single archive file in a well-defined format (ex. NuGet) that contain files appropriate for the package type (ex. DLL, PDB).[21] Many artifacts result from builds but other types are crucial as well. Packages are essentially one of two things: a library or an application.[22]

Compared to source files, binary artifacts are often larger by orders of magnitude, they are rarely deleted or overwritten (except for rare cases such as snapshots or nightly builds), and they are usually accompanied by much metadata such as id, package name, version, license and more.

Metadata

Metadata describes a binary artifact, is stored and specified separately from the artifact itself, and can have several additional uses. The following table shows some common metadata types and their uses:

Metadata typeUsed for
Versions availableUpgrading and downgrading automatically
DependenciesSpecify other artifacts that the current artifact depends on
Downstream dependenciesSpecify other artifacts that depend on the current artifact
LicenseLegal compliance
Build date and timeTraceability
DocumentationProvide offline availability for contextual documentation in IDEs
Approval informationTraceability
MetricsCode coverage, compliance to rules, test results
User-created metadataCustom reports and processes


Products providing repository management

Software to manage repositories (repository managers) includes:

  • Apache Archiva – "repository management software [for a] build artifact repository"[23]
  • Inedo's ProGet – "Universal Package Manager. World-class features. Accessible for everyone."[24]
  • JFrog's Artifactory[25][26]
  • Package Drone – "a package manager repository for OSGi"[27]
  • Sonatype's Nexus – : works with build tools like Ant, Ivy, Gradle, Maven, SBT among others.[28]
  • Pulp – "free and open source platform for managing repositories of software packages and making it available to large numbers of consumers. Supported types: RPM, Python, Puppet, Docker and OSTree."[29]

Web sites providing software repositories
  • CloudRepo – "Fully managed, cloud based, private and public repositories."[31]
  • Cloudsmith – "The new standard in Package Management and Software Distribution."[32]
  • Dist – "Reliable, secure, private, and fast Docker Container Registries and Maven Repositories hosted in the cloud."[33]
  • feedz.io – "Package Hosting and Distribution"[34]
  • Github Package Registry[35]
  • MyGet – "continuous delivery service hosting 1000s of NuGet, Bower and NPM package repositories"[36]
  • Packagecloud – "A unified, developer friendly interface for all of your artifacts."[37]
gollark: APIONET is 192841894612841246274619284126581296184509124826820185930573905472395827960367089457209857239058390486265235 security.
gollark: I'm sure you'd like to think so.
gollark: fish has such excellent history search that they are unnecessary.
gollark: I don't use aliases myself.
gollark: `xbps-query -Rs`?

See also

References
  1. itmWEB: Coping with Computer Viruses Archived October 14, 2007, at the Wayback Machine
  2. "The Haskell Cabal | Overview". www.haskell.org. Retrieved 2019-03-25.
  3. "Maven – Welcome to Apache Maven". maven.apache.org. Retrieved 2019-03-25.
  4. "Julia Package Listing". pkg.julialang.org. Retrieved 2019-03-25.
  5. "Quicklisp beta". www.quicklisp.org. Retrieved 2019-03-25.
  6. karann-msft. "NuGet Package Manager UI Reference". docs.microsoft.com. Retrieved 2019-03-25.
  7. "npm". www.npmjs.com. Retrieved 2019-03-25.
  8. "Installing Perl Modules - www.cpan.org". www.cpan.org. Retrieved 2019-03-25.
  9. Leisch, Friedrich. "Creating R Packages: A Tutorial" (PDF).
  10. Graves, Spencer B.; Dorai-Raj, Sundar. "Creating R Packages, Using CRAN, R-Forge, And Local R Archive Networks And Subversion (SVN) Repositories" (PDF).
  11. "The Comprehensive R Archive Network". cran.r-project.org. Retrieved 2019-03-25.
  12. "R Installation and Administration". cran.r-project.org. Retrieved 2019-03-25.
  13. "R-Forge: Welcome". r-forge.r-project.org. Retrieved 2019-03-25.
  14. "The Cargo Book". Documentation. Rust Programming Language. Retrieved 2019-08-26.
  15. "Rust Package Registry". crates.io. Retrieved 2019-08-26.
  16. "List of Top Repositories by Programming Language". Stack Overflow. Retrieved 2010-04-14.
  17. "npm About". www.npmjs.com. Retrieved 2019-11-21.
  18. developers, The pip, pip: The PyPA recommended tool for installing Python packages., retrieved 2019-11-21
  19. "Apt - Debian Wiki". wiki.debian.org. Retrieved 2019-11-22.
  20. "Homebrew". Homebrew. Retrieved 2019-11-22.
  21. Chris, Tucker (2007-03-15). "Optimal Package Install/Uninstall Manager" (PDF). UC San Diego: 1. Retrieved 2011-09-14. Cite journal requires |journal= (help)
  22. "Linux repository classification schemes". braintickle.blogspot.com. Retrieved 2008-03-01.
  23. "Apache Archiva: The Build Artifact Repository Manager". The Apache Software Foundation. Retrieved 2013-04-17. Apache Archiva[...] is an extensible repository management software that helps taking care of your own personal or enterprise-wide build artifact repository.
  24. "ProGet". Inedo. Retrieved 2016-02-11. Consistency, continuity, compliance – all in one centralized universal package manager with ProGet.
  25. "Jfrog Artifactory". Wikieduonline. JFrog Artifactory is a binary repository manager software designed to store the binary output of the build process for use in distribution and deployment
  26. "Artifactory. Manage Your Binaries". JFrog. Retrieved 2014-10-20. As the first Binary Repository Management solution, Artifactory has changed the way binaries are controlled, stored and managed throughout the software release cycle.
  27. "Package Drone". Retrieved 2015-01-23. The idea is to have a workflow of Tycho Compile -> publish to repo -> Tycho Compile (using deployed artifacts). And some repository tools like cleanup, freezing, validation.
  28. "Nexus Repository Manager". Sonatype. Retrieved 2014-05-21. Nexus Pro gives you more information, more control, and better collaboration across your team than ever before. And it works with build tools like Ant, Ivy, Gradle, Maven, SBT and others. Use Nexus as the foundation for your complete Component Lifecycle Management approach.
  29. "Pulp | software repository management". pulpproject.org. Retrieved 2017-07-11.
  30. "Azure Artifacts". Azure Artifacts
  31. "CloudRepo". CloudRepo. Retrieved 2017-01-01. CloudRepo is a cloud native artifact repository manager offering both public and private repositories. CloudRepo allows high performance software development teams to securely store and share artifacts for use in other builds and development processes.
  32. "Cloudsmith". Cloudsmith. Retrieved 2020-05-20. Cloudsmith is the preferred software platform for securely storing and sharing packages and containers. We have distributed millions of packages for innovative companies around the world.
  33. "Dist". Dist. Retrieved 2020-03-04. Develop and deploy with confidence. Your artifacts are a critical part of your development and deployment pipelines. Dist is designed to be highly available to match your production needs, and fast to keep your developers happy and productive.
  34. "Package Hosting and Distribution". feedz.io. Retrieved 2020-02-10. Store and distribute your private NuGet and npm packages with no user limit and transparent pricing
  35. https://github.blog/2019-05-10-introducing-github-package-registry/
  36. "MyGet: Hosted NuGet, NPM, Bower and Vsix". MyGet. Retrieved 2013-03-13. MyGet hosts thousands of NuGet, Bower and NPM repositories used by companies and individual developers worldwide. MyGet comes with built-in Build Services, and also provides friction-free integration with GitHub, BitBucket and Visual Studio Online.
  37. Canals, Armando (2018-06-25). "Continuous package publishing, part I: introduction to package management in CI/CD". circleci.com. [packagecloud] hosts private and public package repositories for many different package types and works seamlessly with different package managers.
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.