Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/jrmarino/synth

Next D/Ports build tool for live systems (Alternative for Portmaster and Portupgrade tools)
https://github.com/jrmarino/synth

Last synced: 20 days ago
JSON representation

Next D/Ports build tool for live systems (Alternative for Portmaster and Portupgrade tools)

Awesome Lists containing this project

README 

        
### Synth -  Custom package repository builder for FreeBSD and DragonFly



Synth is an advanced concurrent (parallel) ports building tool aimed at

regular users that prefer or require building their own packages from

source.  Synth will build packages in a clean environment that exactly

mirrors the system it builds on, creates a local repository, installs a

pkg repository configuration file that causes the local packages to be

used with the highest priority, and automatically upgrades the system

with a single command.



#### > synth version



The version screen appears when Synth is executed with no command, an

unknown command, or the "version" command.  It displays the current version,

the copyright, and a summary of valid commands along with the usage.



    ====================================================================

      Custom package repository builder for FreeBSD and DragonFly 2.02

    ====================================================================

                   Copyright (C) 2015-2017 John R. Marino

    

    

    Usage: synth [zero-parameter-option]

    -or-   synth [list-option] 

    

    zero-parameter-option includes: help, configure, version, status,

                          upgrade-system, prepare-system, status-everything

                          everything, purge-distfiles, rebuild-repository

    list-option includes: status, build, just-build, install, force, test



#### > synth help



The help screen appears when "synth help" is entered from the command line.

It provides a slightly longer explanation of each command Synth recognizes.

For the most detailed explanation of the Synth commands, please refer the

manual page, e.g. "man synth".



    Summary of command line options - see synth.1 man page for more details

    ===============================================================================

    synth status              Dry-run: Shows what 'upgrade-system' would build

    synth configure           Brings up interactive configuration menu

    synth upgrade-system      Incremental rebuild of installed packages on system.

                              Afterwards, the local repository is rebuilt and the

                              system packages are automatically upgraded.

    synth prepare-system      Like 'upgrade-system' but ends when repo is rebuilt

    synth rebuild-repository  Rebuilds local Synth repository on command

    synth purge-distfiles     Deletes obsolete source distribution files

    synth status-everything   Dry-run: Shows what 'everything' would build

    synth everything          Builds entire ports tree and rebuilds repository

    synth version             Displays version, description and usage summary

    synth help                Displays this screen

    synth status [ports]      Dry-run: Shows what will be rebuilt with given list

    synth build [ports]       Incrementally build ports based on given list, but

                              asks before updating repository and system

    synth just-build [ports]  Like 'build', but skips post-build questions

    synth install [ports]     Like 'build', but upgrades system without asking

    synth force [ports]       Like 'build', but deletes existing packages first

    

    synth test [ports]        Just builds with DEVELOPER=yes; pre-deletes pkgs

    

    [ports] is a space-delimited list of origins, e.g. editors/joe editors/emacs.

    It may also be a path to a file containing one origin per line.



There are two command forms: commands with no arguments (e.g. help, version)

and that take a list of port origins (unlimited) or a single path to file

containing lists of port origins, one per line.  These are equivalent:



```

> synth just-build editors/joe editors/nano editors/libreoffice

```



```

> synth just-build /tmp/build.list   

```



(where the file /tmp/build.list contains:

```

editors/joe

editors/nano

editors/libreoffice

```



#### > synth configure



Synth uses an interactive menu to configure itself.  It is based on profiles

that can mix and match the parts of the build system.  Most people will use

the "LiveSystem" profile, but advanced users may have custom bases installed

elsewhere and prefer to create a profile that uses them.  The configuration

also covers items such as how many builders to spawn during building, whether

tmpfs should be used, etc.



    Synth configuration profile: LiveSystem

    ===============================================================================

       [A] Ports directory            /usr/ports

       [B] Packages directory         /var/synth/live_packages

       [C] Distfiles directory        /usr/ports/distfiles

       [D] Port options directory     /var/db/ports

       [E] Build logs directory       /var/log/synth

       [F] Build base directory       /usr/obj/synth-live

       [G] System root directory      /

       [H] Compiler cache directory   disabled

       [I] Num. concurrent builders   6

       [J] Max. jobs per builder      4

       [K] Use tmpfs for work area    true

       [L] Use tmpfs for localbase    true

       [M] Display using ncurses      true

       [N] Fetch prebuilt packages    false

    

       [>]   Switch/create profiles

       [RET] Exit

    

    Press key of selection:



Just press the letter of the item that needs configuring.  All changes will be

marked with an asterisk, and pressing the Enter key (carriage return) will

save the changes.



#### > synth status

```

> synth status

> synth status [ports]

> synth status-everything

```



The "Status" command is unique in that there are three versions of it, and it has

both the singular and ports-list form.  The purpose of the "Status" command is to

perform a "dry-run" of the intended command without actually changing anything on

the system.  It will show which packages will be deleted due to failed validity

checks, and which packages will be built as a result.  The "synth status" command

looks at the current list of installed packages and checks the given ports tree.

From that it calculates which ones are outdated, and how far the rebuild will

cascade.  The result is the number of packages that will be rebuilt to bring

the current packages up to date.  The "synth status [ports]" accepts a list of

ports and checks them against the local repository, and from that determines

what actually will be rebuilt.  The "synth status-everything" command returns

the incremental list of ports that would be built if the entire ports tree is

requested (this is obviously not a command for an average user).



    # synth status databases/postgresql96-server

    These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):

      N => databases/postgresql96-client

      N => devel/icu

      N => databases/postgresql96-server

    Total packages that would be built: 3

    The complete build list can also be found at:

    /var/synth/synth_status_results.txt



#### > synth upgrade-system



This is a popular command.  It would be used after bringing the ports tree up

to date. Executing "synth upgrade-system" will perform the same analysis as

"synth status", but then starting building all the required ports concurrently.

When that is finished, the local repository will be updated, and finally the

system's pkg(8) program will be commanded to update the system using the local

repository.



![synth status](www/synth_ncurses.png)



During the build process, the build status is shown and updated every second.

The user can see the results of recent builds (up to 50, but limited to screen

size), the status of each builder, and some statistics, include system load,

swap status, and the average number of packages built per hour.  The "impulse"

statistic is the package build rate over the last 500 seconds.  The "Lines"

column in the builder section is the current length of the build log, updated

each second.



#### > synth prepare-system



This command is similar to the "synth upgrade-system" command except that it

stops after rebuilding the local repository.  The sysadmin can then upgrade

the system using pkg(8) at their leisure.



#### > synth rebuild-repository



This command will do a sanity check on all built packages and remove the

invalid ones.  It will then (re)create a local repository comprised of the

packages that remain.



#### > synth just-build [ports]



This command will build the list of given ports and when it has finished, the

program will provide a final tally of the results and exit.



#### > synth build [ports]



This command will build the list of given ports, and when it has finished, it

will prompt the user to answer "T" (True) or "F" (False) if they want to

rebuild the local repository.  Just the pre-scanning can take a few minutes so

normally the answer would be "F" until the user believes the last package has

been built.  After the repository is rebuilt, the user will be asked another

T/F question to confirm they want to install the specifically listed ports on

to the system.



#### > synth install [ports]



This command is the same as "synth build [ports]" except it will not ask any

questions.  When the build is finished, it rebuilds the repository and

installs the ports automatically.



#### > synth force [ports]



This command is similar to "synth build [ports]" but the difference is that

any package that exists for the listed ports is deleted first, even if it is

a perfectly valid and up-to-date package.  The result of a "synth build"

command could be that nothing gets built, but "synth force" will always build

what it requested.



#### > synth test [ports]



This command is similar to "synth force [ports]", but the difference is that

the ports are built under DEVELOPER mode and have additional checks.  This

makes the build logs suitable for submitting to FreeBSD Bugzilla (as an

alternative to Poudriere)



#### > synth everything



The vast majority of people will not need the "synth everything" command.

It builds every package in the ports tree and rebuilds the local repository

(without asking) when it's done.



#### > synth purge-distfiles



The command will remove all previously fetched distfiles that are no longer

referenced by the ports tree.  It takes a few minutes to scan everything

and then deletes the files en-masse without asking.



### Miscellaneous



#### Custom make.conf



Any file that matches the path "/usr/local/etc/synth/[profile]-make.conf"

where [profile] is the name of the selected configuration profile will be

appended to the builders stock make.conf.  Note that the default profile

name is "LiveSystem", so that would make the path is

"/usr/local/etc/synth/LiveSystem-make.conf" for most users.



#### Graceful exits



Do not hit Control-C during the build!  If you want to stop building, hit

the Control-Q combination.  Synth will exit as soon as it can.  If it's hit

during the building process, it will finish the packages that are currently

building but it won't start any new ones.



#### Port Options



Synth uses cached options if they have been saved.  Synth will scan these

options file before starting a build, and if any are obsolete (number of

options don't match the current port, the option names are different, etc)

then it will print out the problematic ports and halt, recommending that

the cached options either be removed or re-saved to something valid.  To

build a package with non-default options, just run

"make -C /usr/ports/[category]/[portname] config" before staring a build.



#### Build logs



Every build produces a log.  By default they are located at

/var/logs/synth, but this location is configurable.  The log name is in the

format [category]___[portname].log.



#### Regular users



Right now Synth can only be executed by the root user



#### Non-curses mode



There is a text mode for the building phase.  It shows much less information

than the curses-based screen, but if curses is acting up, the text mode gets

the job done just fine.  Also, when cron launches Synth, it is required that

it be using a profile that has disabled the curses display.  Alternatively,

curses will be disabled if **TERM=dumb** is set in the environment, which is

a suitable approach for cron jobs.



#### Web interface



Starting with version 1.60, a dynamic web report is automatically generated

for every build.  The report is created in the _Report_ subdirectory of the

logs directory (/var/log/synth/Report/index.html by default).



![web report](http://downloads.dragonlace.net/misc/synth-img/synth_www_3.png)



The report is updated 10 times per minute.  The entire build history is

retained and searchable during the run; it's reset upon subsequent runs.

In addition to the standard search and navigation options, several areas

of the report are clickable.  Click on the *Built*, *Failed*, *Ignored*,

and *Skipped* fields to click filter for results.  Clicking on the *Total*

field will remove all search filters.  Additionally, every cell in the

*No.* column will filter the history for the port origin, which is

particularly useful for finding all the ports that the builders skipped

due to build failures or finding the port set to IGNORE.  The builder ID

column cells also trigger a quick filter for all work performed by the builder.



To view the report on a localhost, simply navigate the browser to the

reports directory.  To view it remotely, a web server has to be installed,

running and configured to expose the Synth logs directory for remote

browsers.



## Frequently Asked Questions



### Synth fails when I run it inside a jail.  What can I do?



First, there is no benefit to running Synth inside a jail.  It internally

creates jail-like environments for each builder and to add another layer

around Synth doesn't provide any extra protection.  It is recommended that

you don't even bother.



Wait, you don't care about that recommendation?  You demand to be able to

run Synth in a jail?  Okay, add the following to jail.conf:



```

enforce_statfs=0

allow.mount

allow.mount.nullfs

allow.mount.tmpfs

allow.mount.devfs

allow.chflags

```

(Courtesy of Dewayne Geraghty)



### Synth can't fetch because I use a proxy server.  How can I fix this?



This requires providing custom environment variables.  Create a file named

/usr/local/etc/synth/-environment (e.g. LiveSystem-environment) and define

one variable per line, e.g. 



```

HTTP_PROXY=http://proxyserver:8888

FTP_PROXY=http://proxyserver:8888



http_proxy=http://proxyserver:8888

ftp_proxy=http://proxyserver:8888

```

Change the urls and port numbers to match your actual proxy server port, of course.



### Synth is configured to prefetch suitable officially-built packages, but very few are actually retrieved.  What's wrong?



This only happens to FreeBSD users.  On FreeBSD, there are basically two official repositories:

_Quarterly_ and _Latest_.  The 10.2 Release and later have pkg(8) configured to use the 

_Quarterly_ packages by default.  Earlier releases are configured to use the _Latest_ packages.



Either repository is fine, but the provided ports tree has to match!  If you decided to continue

with the _Quarterly_ branch (a fine choice if you don't like constant rebuilding) then you need

to provide Synth with a SVN version of the ports tree set to the same _Quarterly_ branch. As could

be deduced from the name, a new SVN branch is created every three months, so Synth users that

choose the _Quarterly_ ports and packages need to remember to switch branches in January, April,

July, and October.



If you want the newest versions of software always and still leverage prebuilt packages, then ensure

pkg(8) is configured for the _Latest_ packages and the ports tree is as well.  Unlikely

_Quarterly_ users, _Latest_ users never have to change the ports tree configuration.



### Can I use Synth to show UPDATING information?



No, but you can use __pkg(8)__ to accomplish this.  Add something like this to your ~/.cshrc file:



```

alias pnotes    'date -v -4w +%Y%m%d | xargs pkg updating --date'

```



Then the next time you log in, the command "pnotes" will display the last 4 weeks worth of 

UPDATING entries based on what is installed on the system.  Note that most entries are for

portmaster and portupgrade users, and that there will be no action for Synth/pkg users for

the great majority of the entries.



(Courtesy of Matt Smith)



### Can redundant cached port options be removed en-masse?



Yes.  It is not uncommon for an existing port options cache to contain

over a 100 redundant files.  This happens when people build ports on a

live system and just hit "ok" when the options dialog appears.  The causes

the options settings to be saved, but this is unnecessary because the

settings are the same as the defaults.  Later this can cause issues when

the port maintainer updates the options and they no longer match the saved

values.  The redundant files can be listed like this:



```

/bin/sh /usr/ports/Tools/scripts/redundant-opt-files.sh

```



To delete all the redundant options, just pipe the output into *rm*



```

/bin/sh /usr/ports/Tools/scripts/redundant-opt-files.sh | xargs rm -rf

```



### How does one configure Synth to use ccache?



First, install ccache:



```

> pkg install ccache 

or

> cd /usr/ports/devel/ccache && make install

```



Check initial configuration:



```

> ccache -s

cache directory                     /root/.ccache

primary config                      /root/.ccache/ccache.conf

secondary config      (readonly)    /usr/local/etc/ccache.conf

cache hit (direct)                     0

cache hit (preprocessed)               0

cache miss                             0

files in cache                         0

cache size                           0.0 kB

max cache size                       5.0 GB

```



Update the maximum cache size:



```

> ccache --max-size=15G

Set cache size limit to 15.0 GB

```



Set the cache location where you want it (e.g. /var/cache/ccache):



```

> ccache --set-config=cache_dir=/var/cache/ccache

```



check configuration again:



```

> ccache -s

cache directory                     /var/cache/ccache

primary config                      /root/.ccache/ccache.conf

secondary config      (readonly)    /usr/local/etc/ccache.conf

cache hit (direct)                     0

cache hit (preprocessed)               0

cache miss                             0

files in cache                         0

cache size                           0.0 kB

max cache size                      15.0 GB

```



Now run __synth configure__, selection option [H], and enter the value of 

_cache directory_ (/var/cache/ccache in this example)



While synth is building, you can run __ccache -s__ command repeatedly in

another terminal to check if the statistics are changing during the build.

If they are, ccache is properly configured.



### How to configure Synth to crossbuild binaries



Synth can be configured to crossbuild packages with a bit of help from

`binmiscctl(1)` and the `qemu-user-static` port. Currently, synth supports

armv6 and aarch64 crossbuilds on FreeBSD. Note that synth also works

natively on FreeBSD/ARM64 (aarch64), but that option is not available for

FreeBSD/ARM (armv6).



The following example demonstrates how to

build crossbuild armv6 from an amd64 host, and uses [QemuUserModeHowTo](https://wiki.freebsd.org/QemuUserModeHowTo)

as a reference:



First, set up a armv6 sysroot that can be used with `chroot(1)`:

```

# SYSROOT=/crossbuild/armv6

# cd /usr/src

# make -j 6 TARGET=arm TARGET_ARCH=armv6 buildworld

# make DESTDIR=$SYSROOT TARGET=arm TARGET_ARCH=armv6 installworld

```



If this is a new sysroot, you will also need to:

```

# make DESTDIR=$SYSROOT TARGET=arm TARGET_ARCH=armv6 distribution

```



On the host environment, build `qemu-user-static` and copy the emulator binary

into a sysroot directory that will be mounted by synth builds, eg `/sbin`.

Register armv6 binaries to use the the emulator when chroot'd.

```

# synth install emulators/qemu-user-static

# cp /usr/local/bin/qemu-arm-static $SYSROOT/sbin

# binmiscctl add armelf \

  --interpreter /sbin/qemu-arm-static \

  --magic "\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00" \

  --mask  "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff" \

  --size 20 --set-enabled

# mount -t devfs devfs ${DESTDIR}/dev

```



The sysroot is now ready to be used, and can be tested:

```

# chroot -u root $SYSROOT

root@helios:/ # file /usr/bin/file

/usr/bin/file: ELF 32-bit LSB executable, ARM, EABI5 version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.0 (1100510), FreeBSD-style, stripped

```



On a new chroot environment, `/var/run/ld-elf.so.hints` needs to be generated:

```

root@helios:/ # service ldconfig start

```



Set up separate log directories and package repos for synth on the host:

```

# mkdir /var/synth/armv6

# mkdir /var/log/synth-armv6

```



Create a new profile, eg: *ArmBuild*, to use when crossbuilding. In

particular, change:

* Packages directory : `/var/synth/armv6`

* Build logs directory : `/var/log/synth-armv6`

* System root directory : `/crossbuild/armv6`



To run the crossbuilding synth from the host environment, you will need to

ensure that correct profile and package repository database is consulted, eg:

```

# env SYNTHPROFILE=ArmBuild PKG_DBDIR="${SYSROOT}/var/db/pkg" synth build x11/xorg

```



Current drawbacks:

* `synth upgrade-system` will not work, as it updates the host, not the crossbuild



(Courtesy of Jonathan Chen)



## Overview Diagrams



![Relationship with ports and pkg(8)](http://downloads.dragonlace.net/misc/synth-img/synth-arch.png)



![Synth configuration summary](http://downloads.dragonlace.net/misc/synth-img/synth-config2.png)



## Significant articles on the Internet



#### Trond's place : [First experiences with ports-mgmt/synth](https://ximalas.info/2017/12/14/first-experiences-with-ports-mgmt-synth/)



## Testimonials



##### Chris_H - FreeBSD forums member - 25 January 2016



> marino@ regarding: 0.98_5 -- smooth as silk!



> Just completed updating my box, using that version. Everything went flawlessly. Failures were handled gracefully, as well as all other events. Well; save the screen corruption, due to (kernel) messages being sent to it. But I didn't see anything in the Changelog, to indicate there were any changes in that regard -- wishful thinking on my part. ;)



> Thank you very much, John. My dev box thanks you, also. :)



##### garry - FreeBSD forums member - 29 January 2016



> I used synth-0.99_2 to build a new repository, reproducing what I had previously built with ports-mgmt/poudriere using the same general setup (3 jails, 3 jobs, tmpfs for the workdir and data). Synth built my 1800 packages in about 18 hours of wall time with no problems. It had taken more than 24 hours of wall time to build that repository with poudriere(8). It seems that Synth is very efficient in setting up and tearing down builders (chroot(8)s / jail(8)s). I noticed that small packages can be done in 20 seconds by Synth but take a minimum of two minutes with poudriere(8).



> Synth is very pleasant to use. Thanks again.



##### fernandel - 2 February 2016



> I am also using Synth from the first day and after all this years which I "spent" with portmaster I am so happy and thankful that is Synth in the ports and for Marinos help too. Thank you.



> BTW: I start synth-upgrade system before bad time and in the morning is everything done



##### Crivens -  Moderator FreeBSD forum - 1 Feb 2016



> I for one, would take the moment to thank you for some great software!



> This thread is turning into rope, sort of thing. Hopefully it will turn into some chapter of the handbook.



##### protocelt - Moderator FreeBSD forum - 2 Feb 2016



> I have 900+ ports installed on my workstation. Last repository rebuild I did, with the ccache(1) cache fully populated, Synth took around 3.5 hours to rebuild the entire repository. I don't know how long Poudriere took on the same machine but I know it was longer. I thought there was a mistake and ports were not getting rebuilt, but after checking, they were all in fact rebuilt. It is quite fast.



##### PacketMan - FreeBSD forums member - 31 Jan / 3 Feb 2016



> Really liking this synth. Thank you so much Marino. :beer:



> Yes [Synth] a great tool; I can't wait to roll it out to all my machines.



##### tankist02 - FreeBSD forums member - 7 March 2016



> @marino Thank a lot for the great tool and explanation what and why it does. I used to have mysterious crashes after running portmaster. These days I use synth exclusively and crashes don't happen anymore. I tried to use poudriere and it was just too much and slow-ish for my simple needs (home desktop).