{"id":20630727,"url":"https://github.com/sysgears/java-berkleydb-queue","last_synced_at":"2025-07-14T09:36:16.888Z","repository":{"id":66053090,"uuid":"88859622","full_name":"sysgears/java-berkleydb-queue","owner":"sysgears","description":"Lightweight fast persistent queue in Java using Berkley DB","archived":false,"fork":false,"pushed_at":"2017-07-14T10:06:31.000Z","size":10,"stargazers_count":11,"open_issues_count":0,"forks_count":4,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-04-15T18:48:46.294Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sysgears.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-04-20T11:48:07.000Z","updated_at":"2024-09-19T04:21:29.000Z","dependencies_parsed_at":"2023-09-01T23:03:01.886Z","dependency_job_id":null,"html_url":"https://github.com/sysgears/java-berkleydb-queue","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sysgears/java-berkleydb-queue","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysgears%2Fjava-berkleydb-queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysgears%2Fjava-berkleydb-queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysgears%2Fjava-berkleydb-queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysgears%2Fjava-berkleydb-queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sysgears","download_url":"https://codeload.github.com/sysgears/java-berkleydb-queue/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysgears%2Fjava-berkleydb-queue/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265269917,"owners_count":23737941,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T14:09:23.461Z","updated_at":"2025-07-14T09:36:16.843Z","avatar_url":"https://github.com/sysgears.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Lightweight fast persistent queue in Java using Berkley DB\n\n[![Twitter Follow](https://img.shields.io/twitter/follow/sysgears.svg?style=social)](https://twitter.com/sysgears)\n\nMy original article has appeared here first:\nhttps://sysgears.com/articles/lightweight-fast-persistent-queue-in-java-using-berkley-db/\n\nI have uploaded the code for my article here to clarify that\nit has Public Domain license.\n\n## Article\n\nRecently I had a task to develop the application which will have large \nwork queue and which need to survive the restarts. The application need \nto be lightweight. After trying several different persistent engines for\nJava I''ve chosen to stick with Berkley DB Java edition. This persistent\nengine is pretty lightweight it is fast, optimized for multi-threaded \nusage and have no problems with reclaiming free space.\n\nAs I needed the fast persistent queue at a cost of possible data loss on\nsystem crash I've chosen non-transactional API for Berkley DB. With \nnon-transactional API the great speed can be achieved for persistent \nqueue at a price of loss of some data at system crash. The more data you\nallow to be lost the greater speed of the queue you will have. Though \nyou can opt to sync to disk each operation on the queue and in that case\nyour data loss will be minimal.\n\n\u003c!--more--\u003e\n\nBerkley DB keeps data sorted by key in B-Tree. By default keys are \nsorted lexicographically byte by byte. But you can override sorting \norder by providing your own comparator. In this implementation of the \nqueue keys are just big integers and sorted in ascending order.\n\nThe DB allows row locking model, this is great for doing multi-threaded \npolling of the queue. But for multi-threaded pushing you either need to \nkeep key counter in separate database or synchronize the push method. \nI've decided to choose later route.\n\nSo, here is the code of the queue with described choices:\n``` java\n\npackage com.sysgears;\n\nimport com.sleepycat.je.*;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.io.Serializable;\nimport java.math.BigInteger;\nimport java.util.Comparator;\n\n/**\n * Key comparator for DB keys\n */\nclass KeyComparator implements Comparator, Serializable {\n\n    /**\n     * Compares two DB keys.\n     *\n     * @param key1 first key\n     * @param key2 second key\n     *\n     * @return comparison result\n     */\n    public int compare(byte[] key1, byte[] key2) {\n        return new BigInteger(key1).compareTo(new BigInteger(key2));\n    }\n}\n\n/**\n * Fast queue implementation on top of Berkley DB Java Edition.\n *\n\n * This class is thread-safe.\n */\npublic class Queue {\n\n    /**\n     * Berkley DB environment\n     */\n    private final Environment dbEnv;\n\n    /**\n     * Berkley DB instance for the queue\n     */\n    private final Database queueDatabase;\n\n    /**\n     * Queue cache size - number of element operations it is allowed to loose in case of system crash.\n     */\n    private final int cacheSize;\n\n    /**\n     * This queue name.\n     */\n    private final String queueName;\n\n    /**\n     * Queue operation counter, which is used to sync the queue database to disk periodically.\n     */\n    private int opsCounter;\n\n    /**\n     * Creates instance of persistent queue.\n     *\n     * @param queueEnvPath   queue database environment directory path\n     * @param queueName      descriptive queue name\n     * @param cacheSize      how often to sync the queue to disk\n     */\n    public Queue(final String queueEnvPath,\n                 final String queueName,\n                 final int cacheSize) {\n        // Create parent dirs for queue environment directory\n        new File(queueEnvPath).mkdirs();\n\n        // Setup database environment\n        final EnvironmentConfig dbEnvConfig = new EnvironmentConfig();\n        dbEnvConfig.setTransactional(false);\n        dbEnvConfig.setAllowCreate(true);\n        this.dbEnv = new Environment(new File(queueEnvPath),\n                                  dbEnvConfig);\n\n        // Setup non-transactional deferred-write queue database\n        DatabaseConfig dbConfig = new DatabaseConfig();\n        dbConfig.setTransactional(false);\n        dbConfig.setAllowCreate(true);\n        dbConfig.setDeferredWrite(true);\n        dbConfig.setBtreeComparator(new KeyComparator());\n        this.queueDatabase = dbEnv.openDatabase(null,\n            queueName,\n            dbConfig);\n        this.queueName = queueName;\n        this.cacheSize = cacheSize;\n        this.opsCounter = 0;\n    }\n\n    /**\n     * Retrieves and returns element from the head of this queue.\n     *\n     * @return element from the head of the queue or null if queue is empty\n     *\n     * @throws IOException in case of disk IO failure\n     */\n    public String poll() throws IOException {\n        final DatabaseEntry key = new DatabaseEntry();\n        final DatabaseEntry data = new DatabaseEntry();\n        final Cursor cursor = queueDatabase.openCursor(null, null);\n        try {\n            cursor.getFirst(key, data, LockMode.RMW);\n            if (data.getData() == null)\n                return null;\n            final String res = new String(data.getData(), \"UTF-8\");\n            cursor.delete();\n            opsCounter++;\n            if (opsCounter \u003e= cacheSize) {\n                queueDatabase.sync();\n                opsCounter = 0;\n            }\n            return res;\n        } finally {\n            cursor.close();\n        }\n    }\n\n    /**\n     * Pushes element to the tail of this queue.\n     *\n     * @param element element\n     *\n     * @throws IOException in case of disk IO failure\n     */\n    public synchronized void push(final String element) throws IOException {\n        DatabaseEntry key = new DatabaseEntry();\n        DatabaseEntry data = new DatabaseEntry();\n        Cursor cursor = queueDatabase.openCursor(null, null);\n        try {\n            cursor.getLast(key, data, LockMode.RMW);\n\n            BigInteger prevKeyValue;\n            if (key.getData() == null) {\n                prevKeyValue = BigInteger.valueOf(-1);\n            } else {\n                prevKeyValue = new BigInteger(key.getData());\n            }\n            BigInteger newKeyValue = prevKeyValue.add(BigInteger.ONE);\n\n            try {\n                final DatabaseEntry newKey = new DatabaseEntry(\n                        newKeyValue.toByteArray());\n                final DatabaseEntry newData = new DatabaseEntry(\n                        element.getBytes(\"UTF-8\"));\n                queueDatabase.put(null, newKey, newData);\n\n                opsCounter++;\n                if (opsCounter \u003e= cacheSize) {\n                    queueDatabase.sync();\n                    opsCounter = 0;\n                }\n            } catch (IOException e) {\n                e.printStackTrace();\n            }\n        } finally {\n            cursor.close();\n        }\n    }\n\n   /**\n     * Returns the size of this queue.\n     *\n     * @return the size of the queue\n     */\n    public long size() {\n        return queueDatabase.count();\n    }\n\n    /**\n     * Returns this queue name.\n     *\n     * @return this queue name\n     */\n    public String getQueueName() {\n        return queueName;\n    }\n\n    /**\n     * Closes this queue and frees up all resources associated to it.\n     */\n    public void close() {\n        queueDatabase.close();\n        dbEnv.close();\n    }\n}\n```\n\nThe unit tests for the queue:\n``` java\n\npackage com.sysgears;\n\nimport org.testng.annotations.Test;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.util.*;\nimport java.util.concurrent.CountDownLatch;\nimport java.util.concurrent.TimeUnit;\n\n@Test\npublic class UTestQueue {\n\n    @Test\n    public void testCreateQueue() {\n        File queueDir = TestUtils.createTempSubdir(\"test-queue\");\n        Queue queue = new Queue(queueDir.getPath(), \"test-queue\", 3);\n        try {\n            assert Arrays.asList(queueDir.listFiles()).contains(new File(queueDir, \"00000000.jdb\"));\n        } finally {\n            queue.close();\n        }\n    }\n\n    @Test public void testPush() throws Throwable {\n        File queueDir = TestUtils.createTempSubdir(\"test-queue\");\n        Queue queue = new Queue(queueDir.getPath(), \"test-queue\", 3);\n        try {\n            queue.push(\"1\");\n            queue.push(\"2\");\n            String head = queue.poll();\n\n            assert head.equals(\"1\");\n        } finally {\n            queue.close();\n        }\n    }\n\n    @Test public void testQueueSurviveReopen() throws Throwable {\n        File queueDir = TestUtils.createTempSubdir(\"test-queue\");\n        Queue queue = new Queue(queueDir.getPath(), \"test-queue\", 3);\n        try {\n            queue.push(\"5\");\n        } finally {\n            queue.close();\n        }\n\n        queue = new Queue(queueDir.getPath(), \"test-queue\", 3);\n        try {\n            String head = queue.poll();\n\n            assert head.equals(\"5\");\n        } finally {\n            queue.close();\n        }\n    }\n\n    @Test public void testQueuePushOrder() throws Throwable {\n        File queueDir = TestUtils.createTempSubdir(\"test-queue\");\n        final Queue queue = new Queue(queueDir.getPath(), \"test-queue\", 1000);\n        try {\n            for (int i = 0; i \u003c 300; i++) {\n                queue.push(Integer.toString(i));\n            }\n\n            for (int i = 0; i \u003c 300; i++) {\n                String element = queue.poll();\n                if (!Integer.toString(i).equals(element)) {\n                    throw new AssertionError(\"Expected element \" + i + \", but got \" + element);\n                }\n            }\n        } finally {\n            queue.close();\n        }\n\n    }\n\n    @Test public void testMultiThreadedPoll() throws Throwable {\n        File queueDir = TestUtils.createTempSubdir(\"test-queue\");\n        final Queue queue = new Queue(queueDir.getPath(), \"test-queue\", 3);\n        try {\n            int threadCount = 20;\n            for (int i = 0; i \u003c threadCount; i++)\n                queue.push(Integer.toString(i));\n\n            final Set set = Collections.synchronizedSet(new HashSet());\n            final CountDownLatch startLatch = new CountDownLatch(threadCount);\n            final CountDownLatch latch = new CountDownLatch(threadCount);\n\n            for (int i = 0; i \u003c threadCount; i++) {\n                new Thread() {\n                    public void run() {\n                        try {\n                            startLatch.countDown();\n                            startLatch.await();\n\n                            String val = queue.poll();\n                            if (val != null) {\n                                set.add(val);\n                            }\n                            latch.countDown();\n                        } catch (Throwable e) {\n                            e.printStackTrace();\n                        }\n                    }\n                }.start();\n            }\n\n            latch.await(5, TimeUnit.SECONDS);\n\n            assert set.size() == threadCount;\n        } finally {\n            queue.close();\n        }\n    }\n\n    @Test public void testMultiThreadedPush() throws Throwable {\n        File queueDir = TestUtils.createTempSubdir(\"test-queue\");\n        final Queue queue = new Queue(queueDir.getPath(), \"test-queue\", 3);\n        try {\n            int threadCount = 20;\n\n            final CountDownLatch startLatch = new CountDownLatch(threadCount);\n            final CountDownLatch latch = new CountDownLatch(threadCount);\n\n            for (int i = 0; i \u003c threadCount; i++) {\n                new Thread(Integer.toString(i)) {\n                    public void run() {\n                        try {\n                            startLatch.countDown();\n                            startLatch.await();\n\n                            queue.push(getName());\n                            latch.countDown();\n                        } catch (Throwable e) {\n                            e.printStackTrace();\n                        }\n                    }\n                }.start();\n            }\n\n            latch.await(5, TimeUnit.SECONDS);\n\n            assert queue.size() == threadCount;\n        } finally {\n            queue.close();\n        }\n    }\n}\n```\n\nAnd here is a main class which measures the queue performance:\n``` java\n\npackage com.sysgears;\n\nimport java.io.File;\n\npublic class Main {\n\n    public static void main(String[] args) throws Throwable {\n        int elementCount = 10000;\n        File queueDir = TestUtils.createTempSubdir(\"test-queue\");\n        final Queue queue = new Queue(queueDir.getPath(), \"test-queue\", 1000);\n        try {\n            long pushStart = System.currentTimeMillis();\n            for (int i = 0; i \u003c elementCount; i++) {\n                queue.push(Integer.toString(i));\n            }\n            long pushEnd = System.currentTimeMillis();\n            System.out.println(\"Time to push \" + elementCount + \" records: \" + (pushEnd - pushStart) + \" ms\");\n\n            long pollStart = System.currentTimeMillis();\n            for (int i = 0; i \u003c elementCount; i++) {\n                String element = queue.poll();\n                if (!Integer.toString(i).equals(element)) {\n                    throw new AssertionError(\"Expected element \" + i + \", but got \" + element);\n                }\n            }\n            long pollEnd = System.currentTimeMillis();\n            System.out.println(\"Time to poll \" + elementCount + \" records: \" + (pollEnd - pollStart) + \" ms\");\n        } finally {\n            queue.close();\n        }\n    }\n}\n```\n\nOutput of running main on my PC:\n\u003cpre\u003e\n\nTime to push 10000 records: 2633 ms\nTime to poll 10000 records: 7764 ms\u003c/pre\u003e\n\nHope this helps,  \nVictor Vlasenko,\nSysGears\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsysgears%2Fjava-berkleydb-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsysgears%2Fjava-berkleydb-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsysgears%2Fjava-berkleydb-queue/lists"}