{"id":15056833,"url":"https://github.com/shathor/gaia-cluster","last_synced_at":"2026-01-02T10:03:46.283Z","repository":{"id":78353324,"uuid":"84662946","full_name":"shathor/Gaia-Cluster","owner":"shathor","description":"Provides a scaffold to easily build a cluster to query the data from ESA's Gaia satellite. Gaia is an ambitious mission to chart a three-dimensional map of our Galaxy, the Milky Way. Gaia will provide unprecedented positional and radial velocity measurements with the accuracies needed to produce a stereoscopic and kinematic census of about one billion stars in our Galaxy and throughout the Local Group. This amounts to about 1 per cent of the Galactic stellar population.","archived":false,"fork":false,"pushed_at":"2017-03-12T00:04:55.000Z","size":190,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-14T07:45:54.859Z","etag":null,"topics":["apache-cassandra","apache-spark","astronomy","big-data","bigdata","cassandra","cluster","distributed-computing","esa","hadoop","java","java-8","machine-learning","map-reduce"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shathor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-03-11T16:15:56.000Z","updated_at":"2024-12-12T17:50:47.000Z","dependencies_parsed_at":"2023-03-12T04:11:41.633Z","dependency_job_id":null,"html_url":"https://github.com/shathor/Gaia-Cluster","commit_stats":{"total_commits":21,"total_committers":2,"mean_commits":10.5,"dds":0.4285714285714286,"last_synced_commit":"f56bd07620624820f59c3b503a4a060df9f1d28a"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shathor%2FGaia-Cluster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shathor%2FGaia-Cluster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shathor%2FGaia-Cluster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shathor%2FGaia-Cluster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shathor","download_url":"https://codeload.github.com/shathor/Gaia-Cluster/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243544665,"owners_count":20308168,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-cassandra","apache-spark","astronomy","big-data","bigdata","cassandra","cluster","distributed-computing","esa","hadoop","java","java-8","machine-learning","map-reduce"],"created_at":"2024-09-24T21:56:47.132Z","updated_at":"2026-01-02T10:03:46.191Z","avatar_url":"https://github.com/shathor.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Gaia-Cluster\n\n![alt text](../master/src/main/resources/gaia.gif \"Gaia\")\n\n## Description\n\nProvides a scaffold to build and develop a cluster to query the data from ESA's Gaia satellite. The cluster uses open-source technology: Apache Spark as the data process engine\nand Apache Cassandra as database.\n\n## Gaia\n\nGaia is an ambitious mission to chart a three-dimensional map of our Galaxy, the Milky Way, in the process revealing the composition, formation and evolution of the Galaxy.\nGaia will provide unprecedented positional and radial velocity measurements with the accuracies needed to produce a stereoscopic and kinematic census of about one billion stars\nin our Galaxy and throughout the Local Group. This amounts to about 1 per cent of the Galactic stellar population.\n[[Source]](http://sci.esa.int/gaia/)\n\n## Gaia-Cluster\n\n### Setup\n\nThis setup focuses on a single node test cluster for your local machine. Using:\n* Java 8\n  * [Oracle](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)\n  * [OpenJDK Windows](https://developers.redhat.com/products/openjdk/download/)\n  * [OpenJDK Linux/Mac](http://openjdk.java.net/install/index.html)\n\n* [Maven](https://maven.apache.org/) - Project and dependency management, build tool\n  * Windows users: Add Maven to PATH environment variables: Run cmd as Administrator \u003e setx /M PATH \"%PATH%;\\\u003cpath-to-Maven-bin-directory\u003e\"\n* [Apache Cassandra](http://cassandra.apache.org/) - A distributed, scalable NoSQL database\n  * Windows users: I recommend installing from [Datastax](https://www.datastax.com/2012/01/getting-started-with-apache-cassandra-on-windows-the-easy-way)\n  * Windows users: Make sure the service is running: Windows + R \u003e services.msc \u003e DataStax DDC Server 3.9.0 \u003e Start\n* [Apache Spark](http://spark.apache.org/) - A fast and general engine for large-scale data processing. Used as dependency, no need for installation.\n* Checkout or download this project\n* Run `mvn clean install` on the top level directory to build it\n* Run `mvn exec:java -D exec.mainClass=GaiaClusterDemo` or `mvn exec:java -D exec.mainClass=DataImport`. Though recommended to import the project in your IDE and to develop and run the main classes from there.\n\n### Data Import\nESA released the first catalogue [Gaia DR 1](https://www.cosmos.esa.int/web/gaia/release) the 14 September 2016.\nThe data is available at their [archive page](https://gea.esac.esa.int/archive/) in form of [CSV](http://cdn.gea.esac.esa.int/Gaia/gaia_source/csv/) amongst other.\n\nThe amount of data is huge. Around  1'142'727'643 rows, 57 columns each.\n\nTo faciliate the data import into Cassandra, there is helper class: [DataImport](../master/src/main/java/DataImport.java). Following steps are executed for all 5231 CSV files until finished:\n\n1. Download the next compressed CSV source file from Gaia archive\n2. Decompress the the CSV source file\n3. Delete compressed CSV source file (no longer needed)\n4. Parse all rows of the CSV source file into memory\n5. Delete decompressed CSV source file (no longer needed)\n6. Insert the rows into the Cassandra database\n\nThe whole data import will take a day, but can be stopped and restarted.\n\n**Important**: The data importer sets up the Cassandra keyspace with SimpleStrategy replication and replacation factor 1. Primary keys are 'solution_id' and 'source_id'.\n\nGiven the program argument \"-restart true\", any previously imported data will be dropped and the data import is started anew. In case the restart argument is set to false or no argument is given,\nthe data import is checking the \"imported.csv\" file for already imported data source files and excludes those. This mode allows the data import to be  interrupted, stopped and started again and\nprovides some fault tolerance.\n \nNote that no checks are made for completeness or data integrity.\n\nFor testing you can also setup the Cassandra database with some test data by using the Cassandra CQL Shell. An example script can be found in the [data folder](../master/src/main/resources/data/cassandra_init.txt).\n\n### Configuration\n\nThe cluster connections are configured in the [application.yaml](../master/src/main/resources/config/application.yaml). It is both relevant for the cluster and the data importer:\n\n``` yaml\nspark:\n  appname: GaiaCluster\n  master: local[4]\n  connectionHost: spark.cassandra.connection.host\n\ncassandra:\n  address: 127.0.0.1\n  keyspace: gaia_cluster\n  gaiasource: gaia_source\n ```\n\n### Executing queries\nThe [GaiaClusterDemo](../master/src/main/java/GaiaClusterDemo.java) shows how to start a Apache Spark context and some examples of querying the data.\n\n#### Starting the Apache Spark context:\n\n``` java\nSparkConf sparkConf = new SparkConf();\nsparkConf.setAppName(Config.spark().appname());\n\nsparkConf.setMaster(Config.spark().master());\nsparkConf.set(Config.spark().connectionHost(), Config.cassandra().address());\ntry(JavaSparkContext sc = new JavaSparkContext(conf)) {\n    // write queries here using \"sc\"\n}\n// after try-with-resources: Apache Spark is stopped\n\t\t\n```\n\n#### Count all data sets:\n``` java\nprivate long selectCount(JavaSparkContext sc) {\n\tString keyspace = Config.cassandra().keyspace();\n\tString table = Config.cassandra().gaiasource();\n\treturn javaFunctions(sc)\n\t\t\t.cassandraTable(keyspace, table, new SolutionRowReaderFactory(Field.SOLUTION_ID)) // using one of the primary keys.\n\t\t\t.count();\n}\n```\n\n#### Specify the fields you want to get data from and/or are involved in the query. Restricting fields to a subset will speedup queries:\n``` java\nList\u003cField\u003c?\u003e\u003e fields = Arrays.asList(Field.REF_EPOCH, Field.MATCHED_OBSERVATIONS, Field.L, Field.B);\n```\n\n#### In case you need all fields of a solution (one complete row):\n``` java\nList\u003cField\u003c?\u003e\u003e fields = Arrays.asList(Field.values());\n```\n\n\n#### Example generic select of fields with a filter function\n``` java\nprivate List\u003cSolution\u003e select(JavaSparkContext sc, List\u003cField\u003c?\u003e\u003e fields, Function\u003cSolution, Boolean\u003e filterFunction) {\n\tString keyspace = Config.cassandra().keyspace();\n\tString table = Config.cassandra().gaiasource();\n\tJavaRDD\u003cSolution\u003e cassandraRdd = javaFunctions(sc)\n\t\t\t.cassandraTable(keyspace, table, new SolutionRowReaderFactory(fields))\n\t\t\t.select(Field.columnNames(fields))\n\t\t\t.filter(filterFunction);\n\treturn cassandraRdd.collect();\n}\n```\n\n#### Example of calling the above `select` method to select all solutions (discouraged on the complete dataset: Would be slow or lead to out-of-memory exception):\n``` java\nprivate List\u003cSolution\u003e selectAll(JavaSparkContext sc) {\n\treturn select(sc, Arrays.asList(Field.values()), (Solution solution) -\u003e true);\n}\n```\n\n#### Example of calling the above `select` method to select solutions that were observed 42 times (never tested on full dataset, might also lead to out-of-memory exception):\n``` java\nprivate List\u003cSolution\u003e selectMatchedObservations42(JavaSparkContext sc, List\u003cField\u003c?\u003e\u003e fields) {\n\tFunction\u003cSolution, Boolean\u003e filterFunction = (Solution solution) -\u003e Objects.equals(solution.getMatchedObservations(), 42);\n\treturn select(sc, fields, filterFunction);\n}\n```\n\nNote that the JavaRDD class by Apache Spark allows all kinds of map-reduce functionality beyond select or filter.\n\n#### Getting the desired values from returned solutions, there are two ways (Logging may only work on local machine):\n``` java\n// calling getters on solution\nfor(Solution solution : solutions) {\n    LOG.info(\"Galactic Latitude: \" + solution.getGalacticLatitude());\n}\n\n// calling get on the desired Field and passing the solution\nfor(Solution solution : solutions) {\n    LOG.info(\"Galactic Latitude: \" + Field.B.get(solution)); // Field.B is the galactic latitude field\n}\n\n// also Field makes generic access possible\nfor(Solution solution : solutions) {\n    for(Field\u003c?\u003e field : fields) {\n        LOG.info(\"Extracted: \" + field.get(solution));\n    }\n}\n\n```\n\nNote that the getters will return null, if they were not selected explicitely in the query (for performance reasons).\n\n#### Inserting solutions into Cassandra (for intermediate results and to persist results in a distributed cluster context):\n``` java\nprivate static void insertSolution(JavaSparkContext sc, List\u003cSolution\u003e solutions) {\n\tJavaRDD\u003cSolution\u003e rdd = sc.parallelize(solutions);\n\n\tSolutionRowWriterFactory factory = new SolutionRowWriterFactory();\n\n\tString keyspace = Config.cassandra().keyspace(); // maybe use a different keyspace\n\tString table = Config.cassandra().gaiasource(); // use a different table !!\n\n\tCassandraJavaUtil.javaFunctions(rdd).writerBuilder(keyspace, table, factory).saveToCassandra();\n}\n```\n\n\n\n\n\n\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshathor%2Fgaia-cluster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshathor%2Fgaia-cluster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshathor%2Fgaia-cluster/lists"}