README.md (20116B)
1 ## The XBPS source packages collection 2 3 This repository contains the XBPS source packages collection to build binary packages 4 for the Void Linux distribution. 5 6 The included `xbps-src` script will fetch and compile the sources, and install its 7 files into a `fake destdir` to generate XBPS binary packages that can be installed 8 or queried through the `xbps-install(1)` and `xbps-query(1)` utilities, respectively. 9 10 See [Contributing](./CONTRIBUTING.md) for a general overview of how to contribute and the 11 [Manual](./Manual.md) for details of how to create source packages. 12 13 ### Table of Contents 14 15 - [Requirements](#requirements) 16 - [Quick start](#quick-start) 17 - [chroot methods](#chroot-methods) 18 - [Install the bootstrap packages](#install-bootstrap) 19 - [Configuration](#configuration) 20 - [Directory hierarchy](#directory-hierarchy) 21 - [Building packages](#building-packages) 22 - [Package build options](#build-options) 23 - [Sharing and signing your local repositories](#sharing-and-signing) 24 - [Rebuilding and overwriting existing local packages](#rebuilding) 25 - [Enabling distcc for distributed compilation](#distcc) 26 - [Distfiles mirrors](#distfiles-mirrors) 27 - [Cross compiling packages for a target architecture](#cross-compiling) 28 - [Using xbps-src in a foreign Linux distribution](#foreign) 29 - [Remaking the masterdir](#remaking-masterdir) 30 - [Keeping your masterdir uptodate](#updating-masterdir) 31 - [Building 32bit packages on x86_64](#building-32bit) 32 - [Building packages natively for the musl C library](#building-for-musl) 33 - [Building void base-system from scratch](#building-base-system) 34 35 ### Requirements 36 37 - GNU bash 38 - xbps >= 0.56 39 - curl(1) - required by `xbps-src update-check` 40 - flock(1) - util-linux 41 - bsdtar or GNU tar (in that order of preference) 42 - install(1) - GNU coreutils 43 - objcopy(1), objdump(1), strip(1): binutils 44 - other common POSIX utilities included by default in almost all UNIX systems. 45 46 `xbps-src` requires a utility to chroot and bind mount existing directories 47 into a `masterdir` that is used as its main `chroot` directory. `xbps-src` supports 48 multiple utilities to accomplish this task: 49 50 - `bwrap` - bubblewrap, see https://github.com/projectatomic/bubblewrap. 51 - `ethereal` - only useful for one-shot containers, i.e docker (used with travis). 52 - `xbps-uunshare(1)` - XBPS utility that uses `user_namespaces(7)` (part of xbps, default). 53 - `xbps-uchroot(1)` - XBPS utility that uses `namespaces` and must be `setgid` (part of xbps). 54 - `proot(1)` - utility that implements chroot/bind mounts in user space, see https://proot-me.github.io/. 55 56 > NOTE: `xbps-src` does not allow building as root anymore. Use one of the chroot 57 methods shown above. 58 59 <a name="quick-start"></a> 60 ### Quick start 61 62 Clone the `void-packages` git repository and install the bootstrap packages: 63 64 ``` 65 $ git clone git://github.com/void-linux/void-packages.git 66 $ cd void-packages 67 $ ./xbps-src binary-bootstrap 68 ``` 69 70 Build a package by specifying the `pkg` target and the package name: 71 72 ``` 73 $ ./xbps-src pkg <package_name> 74 ``` 75 76 Use `./xbps-src -h` to list all available targets and options. 77 78 To build packages marked as 'restricted', modify `etc/conf`: 79 80 ``` 81 $ echo XBPS_ALLOW_RESTRICTED=yes >> etc/conf 82 ``` 83 84 Once built, the package will be available in `hostdir/binpkgs` or an appropriate subdirectory (e.g. `hostdir/binpkgs/nonfree`). To install the package: 85 86 ``` 87 # xbps-install --repository hostdir/binpkgs <package_name> 88 ``` 89 90 Alternatively, packages can be installed with the `xi` utility, from the `xtools` package. `xi` takes the repository of the current working directory into account. 91 92 ``` 93 # xi <package_name> 94 ``` 95 96 <a name="chroot-methods"></a> 97 ### chroot methods 98 99 #### xbps-uunshare(1) (default) 100 101 This utility requires these Linux kernel options: 102 103 - CONFIG\_NAMESPACES 104 - CONFIG\_IPC\_NS 105 - CONFIG\_UTS\_NS 106 - CONFIG\_USER\_NS 107 108 This is the default method, and if your system does not support any of the required kernel 109 options it will fail with `EINVAL (Invalid argument)`. 110 111 #### xbps-uchroot(1) 112 113 This utility requires these Linux kernel options: 114 115 - CONFIG\_NAMESPACES 116 - CONFIG\_IPC\_NS 117 - CONFIG\_PID\_NS 118 - CONFIG\_UTS\_NS 119 120 Your user must be added to a special group to be able to use `xbps-uchroot(1)` and the 121 executable must be `setgid`: 122 123 # chown root:<group> xbps-uchroot 124 # chmod 4750 xbps-uchroot 125 # usermod -a -G <group> <user> 126 127 > NOTE: by default in void you shouldn't do this manually, your user must be a member of 128 the `xbuilder` group. 129 130 To enable it: 131 132 $ cd void-packages 133 $ echo XBPS_CHROOT_CMD=uchroot >> etc/conf 134 135 If for some reason it's erroring out as `ERROR clone (Operation not permitted)`, check that 136 your user is a member of the required `group` and that `xbps-uchroot(1)` utility has the 137 proper permissions and owner/group as explained above. 138 139 #### proot(1) 140 141 The `proot(1)` utility implements chroot and bind mounts support completely in user space, 142 and can be used if your Linux kernel does not have support for namespaces. See https://proot-me.github.io/. 143 for more information. 144 145 To enable it: 146 147 $ cd void-packages 148 $ echo XBPS_CHROOT_CMD=proot >> etc/conf 149 150 <a name="install-bootstrap"></a> 151 ### Install the bootstrap packages 152 153 There is a set of packages that makes up the initial build container, called the `bootstrap`. 154 These packages are installed into the `masterdir` in order to create the container. 155 156 The primary and recommended way to set up this container is using the `binary-bootstrap` 157 command. This will use pre-existing binary packages, either from remote `xbps` repositories 158 or from your local repository. 159 160 There is also the `bootstrap` command, which will build all necessary `bootstrap` packages from 161 scratch. This is usually not recommended, since those packages are built using your host system's 162 toolchain and are neither fully featured nor reproducible (your host system may influence the 163 build) and thus should only be used as a stage 0 for bootstrapping new Void systems. 164 165 If you still choose to use `bootstrap`, use the resulting stage 0 container to rebuild all 166 `bootstrap` packages again, then use `binary-bootstrap` (stage 1) and rebuild the `bootstrap` 167 packages once more (to gain stage 2, and then use `binary-bootstrap` again). Once you've done 168 that, you will have a `bootstrap` set equivalent to using `binary-bootstrap` in the first place. 169 170 Also keep in mind that a full source `bootstrap` is time consuming and will require having an 171 assortment of utilities installed in your host system, such as `binutils`, `gcc`, `perl`, 172 `texinfo` and others. 173 174 ### Configuration 175 176 The `etc/defaults.conf` file contains the possible settings that can be overridden 177 through the `etc/conf` configuration file for the `xbps-src` utility; if that file 178 does not exist, will try to read configuration settings from `$XDG_CONFIG_HOME/xbps-src.conf`, `~/.config/xbps-src.conf`, `~/.xbps-src.conf`. 179 180 If you want to customize default `CFLAGS`, `CXXFLAGS` and `LDFLAGS`, don't override 181 those defined in `etc/defaults.conf`, set them on `etc/conf` instead i.e: 182 183 $ echo 'XBPS_CFLAGS="your flags here"' >> etc/conf 184 $ echo 'XBPS_LDFLAGS="your flags here"' >> etc/conf 185 186 Native and cross compiler/linker flags are set per architecture in `common/build-profiles` 187 and `common/cross-profiles` respectively. Ideally those settings are good enough by default, 188 and there's no need to set your own unless you know what you are doing. 189 190 #### Virtual packages 191 192 The `etc/defaults.virtual` file contains the default replacements for virtual packages, 193 used as dependencies in the source packages tree. 194 195 If you want to customize those replacements, copy `etc/defaults.virtual` to `etc/virtual` 196 and edit it accordingly to your needs. 197 198 <a name="directory-hierarchy"></a> 199 ### Directory hierarchy 200 201 The following directory hierarchy is used with a default configuration file: 202 203 /void-packages 204 |- common 205 |- etc 206 |- srcpkgs 207 | |- xbps 208 | |- template 209 | 210 |- hostdir 211 | |- binpkgs ... 212 | |- ccache ... 213 | |- distcc-<arch> ... 214 | |- repocache ... 215 | |- sources ... 216 | 217 |- masterdir 218 | |- builddir -> ... 219 | |- destdir -> ... 220 | |- host -> bind mounted from <hostdir> 221 | |- void-packages -> bind mounted from <void-packages> 222 223 224 The description of these directories is as follows: 225 226 - `masterdir`: master directory to be used as rootfs to build/install packages. 227 - `builddir`: to unpack package source tarballs and where packages are built. 228 - `destdir`: to install packages, aka **fake destdir**. 229 - `hostdir/ccache`: to store ccache data if the `XBPS_CCACHE` option is enabled. 230 - `hostdir/distcc-<arch>`: to store distcc data if the `XBPS_DISTCC` option is enabled. 231 - `hostdir/repocache`: to store binary packages from remote repositories. 232 - `hostdir/sources`: to store package sources. 233 - `hostdir/binpkgs`: local repository to store generated binary packages. 234 235 <a name="building-packages"></a> 236 ### Building packages 237 238 The simplest form of building package is accomplished by running the `pkg` target in `xbps-src`: 239 240 ``` 241 $ cd void-packages 242 $ ./xbps-src pkg <pkgname> 243 ``` 244 245 When the package and its required dependencies are built, the binary packages will be created 246 and registered in the default local repository at `hostdir/binpkgs`; the path to this local repository can be added to 247 any xbps configuration file (see xbps.d(5)) or by explicitly appending them via cmdline, i.e: 248 249 $ xbps-install --repository=hostdir/binpkgs ... 250 $ xbps-query --repository=hostdir/binpkgs ... 251 252 By default **xbps-src** will try to resolve package dependencies in this order: 253 254 - If a dependency exists in the local repository, use it (`hostdir/binpkgs`). 255 - If a dependency exists in a remote repository, use it. 256 - If a dependency exists in a source package, use it. 257 258 It is possible to avoid using remote repositories completely by using the `-N` flag. 259 260 > The default local repository may contain multiple *sub-repositories*: `debug`, `multilib`, etc. 261 262 <a name="build-options"></a> 263 ### Package build options 264 265 The supported build options for a source package can be shown with `xbps-src show-options`: 266 267 $ ./xbps-src show-options foo 268 269 Build options can be enabled with the `-o` flag of `xbps-src`: 270 271 $ ./xbps-src -o option,option1 pkg foo 272 273 Build options can be disabled by prefixing them with `~`: 274 275 $ ./xbps-src -o ~option,~option1 pkg foo 276 277 Both ways can be used together to enable and/or disable multiple options 278 at the same time with `xbps-src`: 279 280 $ ./xbps-src -o option,~option1,~option2 pkg foo 281 282 The build options can also be shown for binary packages via `xbps-query(1)`: 283 284 $ xbps-query -R --property=build-options foo 285 286 > NOTE: if you build a package with a custom option, and that package is available 287 in an official void repository, an update will ignore those options. Put that package 288 on `hold` mode via `xbps-pkgdb(1)`, i.e `xbps-pkgdb -m hold foo` to ignore updates 289 with `xbps-install -u`. Once the package is on `hold`, the only way to update it 290 is by declaring it explicitly: `xbps-install -u foo`. 291 292 Permanent global package build options can be set via `XBPS_PKG_OPTIONS` variable in the 293 `etc/conf` configuration file. Per package build options can be set via 294 `XBPS_PKG_OPTIONS_<pkgname>`. 295 296 > NOTE: if `pkgname` contains `dashes`, those should be replaced by `underscores` 297 i.e `XBPS_PKG_OPTIONS_xorg_server=opt`. 298 299 The list of supported package build options and its description is defined in the 300 `common/options.description` file or in the `template` file. 301 302 <a name="sharing-and-signing"></a> 303 ### Sharing and signing your local repositories 304 305 To share a local repository remotely it's mandatory to sign it and the binary packages 306 stored on it. This is accomplished with the `xbps-rindex(1)` utility. 307 308 First a RSA key must be created with `openssl(1)` or `ssh-keygen(1)`: 309 310 $ openssl genrsa -des3 -out privkey.pem 4096 311 312 or 313 314 $ ssh-keygen -t rsa -b 4096 -m PEM -f privkey.pem 315 316 > Only RSA keys in PEM format are currently accepted by xbps. 317 318 Once the RSA private key is ready you can use it to initialize the repository metadata: 319 320 $ xbps-rindex --sign --signedby "I'm Groot" --privkey privkey.pem $PWD/hostdir/binpkgs 321 322 And then make a signature per package: 323 324 $ xbps-rindex --sign-pkg --privkey privkey.pem $PWD/hostdir/binpkgs/*.xbps 325 326 > If --privkey is unset, it defaults to `~/.ssh/id_rsa`. 327 328 If the RSA key was protected with a passphrase you'll have to type it, or alternatively set 329 it via the `XBPS_PASSPHRASE` environment variable. 330 331 Once the binary packages have been signed, check the repository contains the appropriate `hex fingerprint`: 332 333 $ xbps-query --repository=hostdir/binpkgs -vL 334 ... 335 336 Each time a binary package is created, a package signature must be created with `--sign-pkg`. 337 338 > It is not possible to sign a repository with multiple RSA keys. 339 340 <a name="rebuilding"></a> 341 ### Rebuilding and overwriting existing local packages 342 343 If for whatever reason a package has been built and it is available in your local repository 344 and you have to rebuild it without bumping its `version` or `revision` fields, it is possible 345 to accomplish this task easily with `xbps-src`: 346 347 $ ./xbps-src -f pkg xbps 348 349 Reinstalling this package in your target `rootdir` can be easily done too: 350 351 $ xbps-install --repository=/path/to/local/repo -yff xbps-0.25_1 352 353 > Please note that the `package expression` must be properly defined to explicitly pick up 354 the package from the desired repository. 355 356 <a name="distcc"></a> 357 ### Enabling distcc for distributed compilation 358 359 Setup the slaves (machines that will compile the code): 360 361 # xbps-install -Sy distcc 362 363 Modify the configuration to allow your local network machines to use distcc (e.g. `192.168.2.0/24`): 364 365 # echo "192.168.2.0/24" >> /etc/distcc/clients.allow 366 367 Enable and start the `distccd` service: 368 369 # ln -s /etc/sv/distccd /var/service 370 371 Install distcc on the host (machine that executes xbps-src) as well. 372 Unless you want to use the host as slave from other machines, there is no need 373 to modify the configuration. 374 375 On the host you can now enable distcc in the `void-packages/etc/conf` file: 376 377 XBPS_DISTCC=yes 378 XBPS_DISTCC_HOSTS="localhost/2 --localslots_cpp=24 192.168.2.101/9 192.168.2.102/2" 379 XBPS_MAKEJOBS=16 380 381 The example values assume a localhost CPU with 4 cores of which at most 2 are used for compiler jobs. 382 The number of slots for preprocessor jobs is set to 24 in order to have enough preprocessed data for other CPUs to compile. 383 The slave 192.168.2.101 has a CPU with 8 cores and the /9 for the number of jobs is a saturating choice. 384 The slave 192.168.2.102 is set to run at most 2 compile jobs to keep its load low, even if its CPU has 4 cores. 385 The XBPS_MAKEJOBS setting is increased to 16 to account for the possible parallelism (2 + 9 + 2 + some slack). 386 387 <a name="distfiles-mirrors"></a> 388 ### Distfiles mirror(s) 389 390 In etc/conf you may optionally define a mirror or a list of mirrors to search for distfiles. 391 392 $ echo 'XBPS_DISTFILES_MIRROR="ftp://192.168.100.5/gentoo/distfiles"' >> etc/conf 393 394 If more than one mirror is to be searched, you can either specify multiple URLs separated 395 with blanks, or add to the variable like this 396 397 $ echo 'XBPS_DISTFILES_MIRROR+=" https://sources.voidlinux.org/"' >> etc/conf 398 399 Make sure to put the blank after the first double quote in this case. 400 401 The mirrors are searched in order for the distfiles to build a package until the 402 checksum of the downloaded file matches the one specified in the template. 403 404 Ultimately, if no mirror carries the distfile, or in case all downloads failed the 405 checksum verification, the original download location is used. 406 407 If you use `proot` or `uchroot` for your XBPS_CHROOT_CMD, you may also specify a local path 408 using the `file://` prefix or simply an absolute path on your build host (e.g. /mnt/distfiles). 409 Mirror locations specified this way are bind mounted inside the chroot environment 410 under $XBPS_MASTERDIR and searched for distfiles just the same as remote locations. 411 412 <a name="cross-compiling"></a> 413 ### Cross compiling packages for a target architecture 414 415 Currently `xbps-src` can cross build packages for some target architectures with a cross compiler. 416 The supported target is shown with `./xbps-src -h`. 417 418 If a source package has been adapted to be **cross buildable** `xbps-src` will automatically build the binary package(s) with a simple command: 419 420 $ ./xbps-src -a <target> pkg <pkgname> 421 422 If the build for whatever reason fails, might be a new build issue or simply because it hasn't been adapted to be **cross compiled**. 423 424 <a name="foreign"></a> 425 ### Using xbps-src in a foreign Linux distribution 426 427 xbps-src can be used in any recent Linux distribution matching the CPU architecture. 428 429 To use xbps-src in your Linux distribution use the following instructions. Let's start downloading the xbps static binaries: 430 431 $ wget http://alpha.de.repo.voidlinux.org/static/xbps-static-latest.<arch>-musl.tar.xz 432 $ mkdir ~/XBPS 433 $ tar xvf xbps-static-latest.<arch>.tar.xz -C ~/XBPS 434 $ export PATH=~/XBPS/usr/bin:$PATH 435 436 If your system does not support `user namespaces`, a privileged group is required to be able to use 437 `xbps-uchroot(1)` with xbps-src, by default it's set to the `xbuilder` group, change this to your desired group: 438 439 # chown root:<group> ~/XBPS/usr/bin/xbps-uchroot.static 440 # chmod 4750 ~/XBPS/usr/bin/xbps-uchroot.static 441 442 Clone the `void-packages` git repository: 443 444 $ git clone git://github.com/void-linux/void-packages 445 446 and `xbps-src` should be fully functional; just start the `bootstrap` process, i.e: 447 448 $ ./xbps-src binary-bootstrap 449 450 The default masterdir is created in the current working directory, i.e `void-packages/masterdir`. 451 452 <a name="remaking-masterdir"></a> 453 ### Remaking the masterdir 454 455 If for some reason you must update xbps-src and the `bootstrap-update` target is not enough, it's possible to recreate a masterdir with two simple commands (please note that `zap` keeps your `ccache/distcc/host` directories intact): 456 457 $ ./xbps-src zap 458 $ ./xbps-src binary-bootstrap 459 460 <a name="updating-masterdir"></a> 461 ### Keeping your masterdir uptodate 462 463 Sometimes the bootstrap packages must be updated to the latest available version in repositories, this is accomplished with the `bootstrap-update` target: 464 465 $ ./xbps-src bootstrap-update 466 467 <a name="building-32bit"></a> 468 ### Building 32bit packages on x86_64 469 470 Two ways are available to build 32bit packages on x86\_64: 471 472 - cross compilation mode 473 - native mode with a 32bit masterdir 474 475 The first mode (cross compilation) is as easy as: 476 477 $ ./xbps-src -a i686 pkg ... 478 479 The second mode (native) needs a new x86 `masterdir`: 480 481 $ ./xbps-src -m masterdir-x86 binary-bootstrap i686 482 $ ./xbps-src -m masterdir-x86 ... 483 484 <a name="building-for-musl"></a> 485 ### Building packages natively for the musl C library 486 487 A native build environment is required to be able to cross compile the bootstrap packages for the musl C library; this is accomplished by installing them via `binary-bootstrap`: 488 489 $ ./xbps-src binary-bootstrap 490 491 Now cross compile `base-chroot-musl` for your native architecture: 492 493 $ ./xbps-src -a x86_64-musl pkg base-chroot-musl 494 495 Wait until all packages are built and when ready, prepare a new masterdir with the musl packages: 496 497 $ ./xbps-src -m masterdir-x86_64-musl binary-bootstrap x86_64-musl 498 499 Your new masterdir is now ready to build packages natively for the musl C library. Try: 500 501 $ ./xbps-src -m masterdir-x86_64-musl chroot 502 $ ldd 503 504 To see if the musl C dynamic linker is working as expected. 505 506 <a name="building-base-system"></a> 507 ### Building void base-system from scratch 508 509 To rebuild all packages in `base-system` for your native architecture: 510 511 $ ./xbps-src -N pkg base-system 512 513 It's also possible to cross compile everything from scratch: 514 515 $ ./xbps-src -a <target> -N pkg base-system 516 517 Once the build has finished, you can specify the path to the local repository to `void-mklive`, i.e: 518 519 # cd void-mklive 520 # make 521 # ./mklive.sh ... -r /path/to/hostdir/binpkgs