https://github.com/difu/arcus

GRIB analysis platform
https://github.com/difu/arcus

aws dwd eccodes grib grib2

Last synced: about 2 months ago
JSON representation

GRIB analysis platform

• Host: GitHub
• URL: https://github.com/difu/arcus
• Owner: difu
• License: mit
• Created: 2017-01-30T19:37:52.000Z (almost 8 years ago)
• Default Branch: master
• Last Pushed: 2019-05-02T16:49:40.000Z (over 5 years ago)
• Last Synced: 2024-07-30T18:13:04.571Z (5 months ago)
• Topics: aws, dwd, eccodes, grib, grib2
• Language: Python
• Homepage:
• Size: 108 KB
• Stars: 3
• Watchers: 3
• Forks: 1
• Open Issues: 1
• Metadata Files:
  • Readme: README.rst
  • License: LICENSE

Awesome Lists containing this project

README

        
============

Arcus

============

Experimental GRIB Cloud Cache and Raster Data query platform

Aim of this project is to evaluate different technologies to analyze, distribute and share GRIB data.

It will use free data of `Deutscher Wetterdienst `_ at ftp://ftp-cdc.dwd.de/pub/REA/COSMO_REA6/

This project uses `eccodes `_ from ECMFW and `gdal `_ for GRIB encoding/decoding.

================================

Quickstart

================================

Infrastructure

""""""""""""""

All infrastructure will be deployed on AWS. To install the AWS command line tools please refer to http://docs.aws.amazon.com/cli/latest/userguide/awscli-install-linux.html.

To create and modify the infrastructure `Terraform `_ is used. Download the ``terraform`` executable and take a look at the `getting started guide `_.

- Create an AWS user and grant this user

  - AWS managed policies

    - SystemAdministrator

    - AmazonElasticFileSystemFullAccess

    - AmazonElasticMapReduceRole

  - Inline policy

  .. code-block:: json

    {

    "Version": "2012-10-17",

    "Statement": [

        {

            "Sid": "Stmt1500631120000",

            "Effect": "Allow",

            "Action": [

                "iam:CreatePolicy",

                "iam:PutRolePolicy",

                "iam:DeleteRolePolicy",

                "iam:CreateRole",

                "iam:AttachRolePolicy",

                "iam:CreateInstanceProfile",

                "iam:AddRoleToInstanceProfile",

                "iam:PassRole",

                "iam:DetachRolePolicy",

                "iam:RemoveRoleFromInstanceProfile",

                "iam:DeleteInstanceProfile",

                "iam:DeleteRole",

                "iam:DeleteUserPolicy",

                "iam:DeletePolicy",

                "elasticmapreduce:RunJobFlow",

                "elasticmapreduce:DescribeCluster",

                "elasticmapreduce:TerminateJobFlows",

                "lambda:AddPermission",

                "lambda:RemovePermission",

                "lambda:PublishLayerVersion",

                "apigateway:*",

            ],

            "Resource": [

                "*"

            ]

        }

    ]

    }

- Create an S3 bucket where Arcus stores its internal components etc. and name it like *my_internal_bucket*. Note that this bucket name must have an unique name. Remember that name as it is needed when you want to deploy the infrastructure.

- Download from Oracle OTN

  - oracle-instantclient18.3-basic-18.3.0.0.0-1.x86_64.rpm

  - oracle-instantclient18.3-sqlplus-18.3.0.0.0-1.x86_64.rpm

  - oracle-instantclient18.3-devel-18.3.0.0.0-1.x86_64.rpm

  and put the files under ``software/oracle/``

- Creation of infrastructure

 ``terraform init``

 ``terraform import aws_s3_bucket.internal_bucket`` *my_internal_bucket_name*

 ``terraform apply -var arcus_internal_bucket_name =`` *my_internal_bucket_name*

For further configuration see the ``variables.tf`` file in the terraform folder.

- Destruction of infrastructure

 ``terraform state rm aws_s3_bucket.internal_bucket``

 ``terraform destroy``