Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/AdguardTeam/FiltersCompiler

A tool that compiles & validates filters
https://github.com/AdguardTeam/FiltersCompiler

adguard filter-list open-source

Last synced: about 2 months ago
JSON representation

A tool that compiles & validates filters

Awesome Lists containing this project

README 

        
# AdGuard Filters Compiler



Filters compiler package is a tool for compiling ad blocking filters into a supported format.

It is used in [FiltersRegistry].



- [Usage](#usage)

- [Tests](#tests)

- [Development](#development)

- [Filters metadata](#filters-metadata)

- [`@include` directive](#include-directive)



### Usage



This package is suggested to be used with filters repository with directory structure presented in tests here.



The package could be run with the following command:



```javascript

const whitelist = [1, 3];

const blacklist = [2];



const path = require('path');

const compiler = require("adguard-filters-compiler");



const filtersDir = path.join(__dirname, './filters');

const logPath = path.join(__dirname, './log.txt');

const reportPath = path.join(__dirname, './report.txt');



const platformsPath = path.join(__dirname, './platforms');



const customPlatformsConfig = {

    // Here you can redefine some of the platforms from platforms.json

    // or add new platforms if you need it.

    "MAC_V3": {

        "platform": "mac",

        "path": "mac_v3",

        "configuration": {

            "ignoreRuleHints": false,

            "removeRulePatterns": [

                "^\\/.*" // remove regex rules for some reason.

            ],

            "replacements": [

                {

                    "from": "regex",

                    "to": "repl"

                }

            ]

        },

        "defines": {

            "adguard": true,

            "adguard_app_mac": true

        }

    },

};



compiler.compile(filtersDir, logPath, reportPath, platformsPath, whitelist, blacklist, customPlatformsConfig);

```



### Tests



```bash

yarn test

```



### Development



In order to add support for new scriptlets and redirects,

you should update `@adguard/tsurlfilter` with updated scriptlets.



For fixing scriptlets converting or validation you should update `scriptlets`.



### Filters metadata



Description of the filters metadata is available in the [FiltersRegistry][filters-metadata] repository.



###  `@include` directive and its options



The `@include` directive provides the ability to include content from the specified address.



#### Syntax:



```text

@include  []

```



where:



- `` — required, URL or same origin relative file path to be included;

- `` — optional, a list of options separated by spaces.

  Available options:



    - `/stripComments` removes AdBlock-style syntax comments from the included file — lines which start with `!`;

    - `/notOptimized` adds a `!+ NOT_OPTIMIZED` hint to the rules;

    - `/exclude=""` excludes from the included file rules

      listed in the exceptions file available by `filepath`;

    - `/addModifiers=""` adds the specified `modifiers` (string as is) to the rules in the included file.

      The addModifiers option can also work with the host-rule format files.

      In this case, host-file comments are to be replaced `#` by AdBlock-style syntax comments `!`;

    - `/ignoreTrustLevel` disables the check of the trust level of the included file.

      Allowed only for the same origin files.

    - `/optimizeDomainBlockingRules` remove redundant rules for domain blocking of the included file.

      Base rules with modifiers and rules of other format will be ignored.

      Comments are not checked separately and may not be relevant after optimization.



> [!IMPORTANT]

> The content of the included file is formatted by the options due to the order of their mention in the directive,

> except `/ignoreTrustLevel`.



#### Examples



- Include a file with domains, add modifiers to the rules, exclude some rules,

   add a hint to the rules, and remove comments from the prepared rules:



    ```adblock

    @include ../input.txt /addModifiers="script" /exclude="../exclusions.txt" /notOptimized /stripComments /optimizeDomainBlockingRules

    ```



    The order of execution of the options is as follows:



    1. `@include ../input.txt`: Includes the content of the file named `input.txt` from the parent directory.



        ``` adblock

        # comment

        example.com

        example.org

        ```



    1. `/addModifiers="script"`: Adds the `$script` modifier to all rules in the included file.



        Result of adding modifiers:



        ``` adblock

        ! comment

        example.com$script

        example.org$script

        ```



        > Used to restrict rules with modifiers when blocking the entire domain would result in a breakage.

        > [issue example](https://github.com/AdguardTeam/FiltersCompiler/issues/190)



    1. `/exclude="../exclusions.txt"`: Excludes rules listed in the exception list from the file named `exclusions.txt`, if they match.



        Due to the content of `exclusions.txt`:



        ``` adblock

        example.com$script

        example2.com$script

        ```



        Result of excluding:



        ``` adblock

        ! comment

        example.org$script

        ```



        > Used to exclude problematic rules in the filter



    1. `/notOptimized`: Adds the `!+ NOT_OPTIMIZED` hint to the rules.



        Result of adding the hint:



        ``` adblock

        ! comment

        !+ NOT_OPTIMIZED

        example.org$script

        ```



        > Used in cases where the filter is designed for mobile site layout and some rules may be removed,

        > due to the lack of ability to collect statistics on mobile platforms.



    1. `/stripComments`: Removes comments in AdBlock style from the included file.



        ``` adblock

        !+ NOT_OPTIMIZED

        example.org$script

        ```



    1. `/optimizeDomainBlockingRules`: Remove only domain blocking redundant rules from the included file.



        Due to the optimization:



        ``` adblock

        ||example.com^

        ||sub.example.com^

        ||domain.com^

        ||test.com^$script

        ```



        Result of optimization:



         ``` adblock

        ||example.com^

        ||domain.com^

        ||test.com^$script

        ```



- Ignore the trust level of the filter list (specified in the metadata) during the file including —

  include the file rules as is:



    ```adblock

    @include ./input.txt /ignoreTrustLevel

    ```



[FiltersRegistry]: https://github.com/AdguardTeam/FiltersRegistry/

[filters-metadata]: https://github.com/AdguardTeam/FiltersRegistry/blob/master/README.md#filters-metadata