Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/foxcapades/eupath-build-plan

WIP EuPathDB Build rework proposal.
https://github.com/foxcapades/eupath-build-plan

Last synced: 5 months ago
JSON representation

WIP EuPathDB Build rework proposal.

Awesome Lists containing this project

README 

          
= Build System

:icons: font

ifdef::env-github[]

:warning-caption: ⚠

endif::[]

:toc: left

Elizabeth Harper 



// External URLS

:gradleDocRoot: https://docs.gradle.org/current/userguide

:condSubstBlog: https://medium.com/codequest/Gradle-multi-project-build-substituting-jar-dependencies-with-local-projects-4a5323f8680b

:condSubstDocs: {gradleDocRoot}/customizing_dependency_resolution_behavior.html

:docSrc: https://github.com/Foxcapades/EuPath-Build-Plan

:docPage: https://Foxcapades.github.io/EuPath-Build-Plan

:javaPlatDocs: https://docs.gradle.org/current/userguide/java_platform_plugin.html

:cpanGitUse: https://metacpan.org/pod/distribution/App-cpanminus/bin/cpanm

:searchRes: {docSrc}/blob/master/search-results

:searchAnt: {searchRes}/ant.txt

:searchMaven: {searchRes}/maven.txt

:gradleDocs: {gradleDocRoot}/userguide.html



Document Render: {docPage} +

Document Source: {docSrc}



== Goals



1. Modernize our build process

2. Break apart the monolith build

3. Simplify/Unify our build scripts

4. Separate build from deployment

5. Separate build from version control



== Increments



=== Part 1: Maven to Gradle



Exchange the maven pom files for <> builds.  Parts of

the Ant build that call to maven will now call to Gradle.



Gradle projects will declare their project dependencies and

the build should function similarly to the previous build

system.



The base-pom file will be converted to a Gradle <> Bill of Materials.



Boilerplate details, such as developer information or custom

<> will be extracted to a

Gradle plugin which can be published to a Maven repository.



This conversion is almost 1 to 1 and wil be tedious, but

should not be not difficult.  This particular change should

only affect users in that the log output will be different.



At this point in time the Gradle distribution will follow

the standard practice of including a base version in the

source repository.



{searchMaven}[All references to Maven] will need to be

removed or updated at this time.



==== Impact on Servers



None as <> will be included with the source code.



For developer convenience (potentially faster builds) it may

be nice to have Gradle installed on the development servers,

but this is not required.



WARNING: TODO: get a list of all non-project scripts that

  call Maven directly



==== Impact on Jenkins



None at this point, the Jenkins `rebuilder` script will call

Ant as normal.



==== Impact on Developers



Different looking log output during build.



Potentially faster project build times.



=== Part 2: Ant to Gradle



Convert the build XML declarations into <> actions.



At this point, custom scripts will be left alone, calls to

them will be maintained, and converted to calls from Gradle.



Most Ant actions or behaviors are available easily in Gradle

or can be easily recreated as a function in Gradle.  Any Ant

functions we rely on which have no built-in mirror in Gradle

will be made into a plugin which will be published to our

internal Maven repository to be made available to all

projects.



{searchAnt}[All references to Ant] will need to be replaced

or updated at this point.



==== Impact on Servers



The `rebuilder` script will be updated to call Gradle rather

than Ant.



WARNING: TODO: get a list of all non-project scripts that

  call ant directly.



==== Impact on Jenkins



The Jenkins `rebuilder` script will be updated to call

Gradle rather than Ant.



==== Impact on Developers



Any Ant calls will now be made through Gradle.



=== Part 3: Scripts to Gradle



Convert the custom wrapper and helper scripts into <> actions.



This will mean custom (Java) code will need to be written.

This code will be created as a Gradle plugin and published

to our internal Maven server so as to be made available to

all project builds.



WARNING: TODO: Get a list of all project wrapper scripts



WARNING: TODO: Get a list of all project calls to those

  wrapper scripts



==== Impact on Servers



WARNING: TODO: Get a list of all non-project wrapper scripts



WARNING: TODO: Get a list of all non-project calls to both

  project and non-project wrapper scripts



==== Impact on Jenkins



WARNING: TODO: figure this out!



==== Impact on Developers



WARNING: Find out what scripts users actually call anymore

  many may be stripped out entirely.



Scripts that users call will be converted to gradle actions.



=== Part 4: Split the monolith



Take each project's <> build and convert its

dependency on sibling projects into artifact dependencies

with <<#condSubst,Conditional Substitution>>.



.Early phase Gradle dependency

[source, groovy]

----

// Require local projects

dependencies {

  compile project(':FgpUtil.Util')

  compile project(':WDK.Model')

  compile project(':WDK.Service')

  ...

}

----



.Late phase Gradle dependency def

[source, groovy]

----

// Use local project if available

configurations.all {

  dependencies.compile { ->

    if it.location.exists {

      substitute module(it.name) with project("${it.group}.${it.name}")

    }

  }

}



// Otherwise default to compiled jar

dependencies {

  compile 'org.gusdb.FgpUtil:Util'

  compile 'org.gusdb.WDK:Model'

  compile 'org.gusdb.WDK:Service'

  ...

}

----



=== Part 5: Deployments



Using a Gradle task on the root projects themselves, a

separate 'deploy' project, or some other tool if we are okay

with duplicating our dependency list, we can build a tar or

tars that bundle the required components for a site. Jenkins

can then copy the tar(s) to the desired location and run any

existing post build steps (such as WDKCache).



[WARNING]

TODO: +

How will conifer fit into this? +

can it be moved to a post step? +

how much work would that be?



== Issues / Concerns



=== Bundling of non-java project code



With <>'s customizable nature, additional

compile tasks can be added for non-Java source code.



In CI additional steps can be performed to publish those

components as bundles or packages as per the standard

practices for the specific language.



In early phases of the build update, non-Java dependencies

will be treated as they are in the current build process.



When we migrate towards a separate build/deploy strategy

we will need to have a system in place to host, version, and

update these dependencies.



Dependencies that do not have a package manager may be

bundled as a OS package for yum or apt, or alternatively

these may just be pulled at deploy time by the deployment

script.



==== Possible approaches by language



===== Perl



With Perl we have a couple of options available to us.  We

can use a private CPAN repository, or alternatively, we can

split the Perl code into separate git repositories and

depend on it {cpanGitUse}[via git itself].



====== Versioning



Depending on the package hosting approach above we may use

different strategies, using git we can use git tags to

maintain releases or different versions.  Using a private

CPAN repository, we would need to come up with a concrete

versioning strategy.



===== JavaScript/TypeScript



This is already handled for us via NPM or Yarn.  Similarly

to Perl, we may desire to host our JavaScript & TypeScript

code in a private NPM repository, or split that code out and

just use git.



==== R



WARNING: TODO



==== C



WARNING: TODO



==== Python



WARNING: TODO



==== Others?



WARNING: TODO



=== Artifact Versioning



WARNING: Everything about this needs to be discussed.  The

  below sections outline a rough idea that may work for our

  team.



==== Releases



For releases, artifacts will be built using the version

format `.` or, for example

`46.0`.  The java platform project will be updated to

reflect the concrete versions of the artifacts that will be

used for a site release.



The `` will be the number of builds of the

release artifact for a site version, so as live patches are

added, this number will increment.



.Example Release History

[source]

----

WDK:

  48.0 - Site Release

  48.1 - Memory leak fix

  49.0 - Site Release

  49.1 - Corrected typo

  49.2 - Fix for user-comment search

----



==== Development



While in development artifacts will be versioned using the

format `.SNAPSHOT-`.



The `` (and potentially ``)

params will be managed by CI to avoid conflicts.



The development <> will be given a

wildcard version identifier that simply specifies that

builds should use the latest artifact.



.Example Version History

[source]

----

WDK:

  46.SNAPSHOT-105: Added public user datasets to user dataset list

  46.SNAPSHOT-106: Corrected issue with deleted public user datasets

--- Site Build 47 released---

  47.SNAPSHOT-1: Added user id field to service endpoint result

  47.SNAPSHOT-2: Reworked question service

----



=== Multi-project development workflow



Initially the projects will declare each other as local

dependencies which will mean that components will be rebuilt

as needed in a manner similar to the previous builds.



Eventually dependencies will be handled with conditional

substitutions, a built in Gradle feature, which will use a

local project when available instead of the Maven repository

version of that dependency.



== Questions



* Depending on different language dependencies?

** These are runtime dependencies and the bundling of the

   deployable package will be managed by the deployment

   script.

+

Perhaps this means we should split repositories in git along

language lines?



* How will conifer fit into this world of separate projects?



* Factoring builds?



* Division of deployments?



* Triggers for builds?  (build cascade when a dependency)

** Does conditional substitution trigger child project builds?



* Should conifer be split out of FgpUtil into it's own project?

** Should all non-java stuff in FgpUtil be split into it's own project?



== Definitions



[#artRep]

=== Artifact Repository



An Artifact Repository is a server or system designed to

provide hosting and versioning for bundled packages of code

or resources for distribution or linking as complete

applications or libraries.



Most widely used languages have one or more commonly used

Package Managers which will generally use an Artifact

Repository, or Source Repository when retrieving or

installing applications or dependencies.



[#condSubst]

=== Conditional Substitution



A Gradle feature that enables the substitution of one

dependency with another if some condition is met.



In our case this would likely mean using a sibling project

instead of an artifact if that project exists in your

workspace.



See {condSubstBlog}[this blog post], or {condSubstDocs}[the

Gradle docs] for a more detailed explanation.



[#gradle]

=== Gradle



Gradle is a build and dependency management and scripting

tool specifically designed around working with JVM

languages.



Gradle's functionality covers the use cases for both Ant and

Maven, and is easily extensible to include our custom build

tasks.



See {gradleDocs}[the Gradle user manual] for more

information.



[#javaPlat]

=== Java Platform



The <> method of creating a Bill of Materials

for a project or group of projects.



The Java Platform can be used to define and/or constrain

projects to a known working set of dependencies.  Similar to

our base-pom files, our dependency versions can be declared

here and omitted from individual project dependency

declarations.



See {javaPlatDocs}[the Gradle docs] for more information.