Backup and Restore of the Elasticsearch Service in AWS S3 bucket

Introduction

This article will help you in taking the manual snapshot of your domain in Amazon Elasticsearch Service. We can easily back up our entire domain this way. You can also take snapshot and restore a single index, or multiple indexes. This blog post walks you through backing up and restoring a single index by using an Amazon S3 bucket.

Steps for Backup of the Elasticsearch Service

  1. Create the s3 bucket
  2. Create the role and attached inline policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::<S3-BUCKET-NAME>"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],

            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::S3-BUCKET-NAME/*"
            ]
        }
    ]
}

3. Update the trust relationship for role.
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "es.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

4. Create the AWS user with secret access key and attached this inline policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::<AWS_ACCOUNT_NUMBER>:role/ROLE-NAME"
        },
        {
            "Effect": "Allow",
            "Action": "es:ESHttpPut",
            "Resource": "arn:aws:es:us-west-2:<AWS_ACCOUNT_NUMBER>:domain/ES-CLUSTER-NAME/*"
        }
    ]
}

5. Install some prerequisites packages in the instance from where you want to run the backup script and update the AWS secret key also from where you running this backup script.
  • yum -y install python-pip
  • pip install requests-aws4auth

6. Update the ES host which you want to take the backup and bucket info.
  • vim /tmp/register-repo.py
import boto3
import requests
from requests_aws4auth import AWS4Auth

host = 'https://search-mb-production-app-us-west-2.es.amazonaws.com/'
region = 'us-west-2' # For example, us-west-1
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

# Register repository
path = '_snapshot/my-snapshot-repo' # the Elasticsearch API endpoint
url = host + path

payload = {
  "type": "s3",
  "settings": {
    "bucket": "elasticsearch-backup-indices",
    "region": "us-west-2",
    "role_arn": "arn:aws:iam::YOUR-ACCOUNT-ID:role/es-s3-backup"
  }
}

headers = {"Content-Type": "application/json"}

r = requests.put(url, auth=awsauth, json=payload, headers=headers)

print(r.status_code)
print(r.text)

7. Execute the file to register repository.
  • chmod 700 /tmp/register-repo.py
  • python /tmp/register-repo.py
Output should be ,
200
{"acknowledged":true}

8. To list all the snapshot from the ES Cluster this is automated snapshot
  • curl -XGET '<Elasticsearch_Endpoint>/_snapshot/cs-automated/_all?pretty'
9. Run this command for taking manual backup.
  • curl -XPUT 'elasticsearch-domain-endpoint/_snapshot/repository/snapshot-name'
10. Use the following command to verify the state of snapshots of your domain:
  • curl -XGET ‘<Elasticsearch_Endpoint>/_snapshot/my-snapshot-repo/_all?pretty'

Steps for Restore the backup data in New ES Cluster(AWS Managed)

1. Create the ES Cluster : List down this all details from this Cluster Info, ES endpoint, Domain ARN, Kibana

2. Data Saved in S3 Bucket : List S3 bucket ARN in which data is stored.
Bucket Info : arn:aws:s3:::<Bucket-name>

3. Create IAM role, Attached one inline policy, Update s3 ARN
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::<Bucket-name>"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::<Bucket-name>/*"
            ]
        }
    ]
}

4. Create one User and attached one inline policy, Update role ARN and ES ARN
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::<AWS_ACCOUNT_NUMBER>:role/<ROLE_NAME>"
        },
        {
            "Effect": "Allow",
            "Action": "es:ESHttpPut",
            "Resource": "arn:aws:es:us-west-2:<AWS_ACCOUNT_NUMBER>:domain/<ES_DOMAIN_NAME>/*"
        }
    ]
}
5. Now Take the AWS Secret access key and update in "aws configure" instance from Where you have to run the script. and update the S3 and ES Cluster info in register-repo.py script.
import boto3
import requests
from requests_aws4auth import AWS4Auth

host = '<ES Endpoint e.g.https://vpc-es-testing-backup-data-g6yblp2q64t3ruenxgyqyztjra.us-west-2.es.amazonaws.com/>'
region = 'us-west-2' # For example, us-west-1
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

# Register repository
path = '_snapshot/my-snapshot-repo' # the Elasticsearch API endpoint
url = host + path

payload = {
  "type": "s3",
  "settings": {
    "bucket": "<BUCKET-NAME>",
    "region": "us-west-2",
    "role_arn": "arn:aws:iam::<AWS_ACCOUNT_NUMBER>:role/<ROLE_NAME"
  }
}

headers = {"Content-Type": "application/json"}

r = requests.put(url, auth=awsauth, json=payload, headers=headers)

print(r.status_code)
print(r.text)

6. Login in elasticsearch head and delete the .kibana indices
7. For restore the data run this command
  • curl -XPOST "<ES ENDOPINT>/_snapshot/my-snapshot-repo/<SNAPSHOT_NAME/_restore?pretty"
for e.g.
  • curl -XPOST "https://vpc-anshu-k-soni-12dsynuqyigf7birwuzortpvcq.us-west-2.es.amazonaws.com/_snapshot/my-snapshot-repo/2019-01-28t06-14-24.35f95d80-ebd7-47d4-b808-6a0ecd587y608/_restore?pretty"









Comments

Post a Comment

Popular posts from this blog

What is the difference between the Roles and Policy in AWS.

In AWS, roles and policies are both integral components of AWS Identity and Access Management (IAM) that govern access and permissions within your AWS resources. However, they serve different purposes and have distinct functionalities. Let's explore the differences between roles and policies: Roles: Purpose: A role is an entity in IAM that defines a set of permissions and trusted entities (such as AWS services or IAM users) that can assume the role. It is used to grant temporary permissions to trusted entities to access AWS resources securely. Use Case: Roles are commonly used to delegate permissions to AWS services or to allow cross-account access. They enable you to grant permissions without sharing long-term access keys and are often utilized by services like EC2 instances, Lambda functions, and other AWS resources. Trust Relationships: Roles have trust relationships that specify the trusted entities that can assume the role. These entities can be IAM users, AWS services, or eve...
Read more

How will you use the ec2 instances if you lost .pem key when you installed first time? How to login now to that ec2 instance.

Losing the .pem key used for the initial setup of an EC2 instance can be a challenging situation, as it is typically the key required to authenticate and access the instance. However, there are a few potential solutions to regain access to the instance: Retrieve the Key Pair from an AMI: If you created an Amazon Machine Image (AMI) from the EC2 instance before losing the .pem key, you can launch a new EC2 instance from that AMI. During the launch process, you can specify a new key pair, allowing you to connect to the new instance using the new key. Mount the Root Volume: Another option is to detach the root volume of the inaccessible instance and attach it as a secondary volume to another running EC2 instance. From there, you can modify the SSH configuration or add the necessary files to regain access to the original instance. Once you have made the modifications, reattach the root volume to the original instance. Use AWS Systems Manager Session Manager: If you have previously set up A...
Read more

Overview of SSL/TLS and encryption

SSL/TLS Protocol: SSL/TLS is a protocol that provides secure communication between clients and servers over the internet. It establishes an encrypted connection to ensure that data transmitted between the client (e.g., web browser) and the server (e.g., website) remains confidential and cannot be tampered with. Encryption: Encryption is the process of encoding data in a way that can only be deciphered by authorized parties who possess the appropriate decryption key. It ensures that sensitive information remains unreadable and protected from unauthorized access during transmission or storage. Key Concepts and Components: Public Key Infrastructure (PKI): PKI is a framework that facilitates the secure exchange of information using public key cryptography. It involves the use of digital certificates, certificate authorities (CAs), and key pairs (public and private keys). Digital Certificates: Digital certificates are electronic documents issued by a trusted third party (CA). They bind an e...
Read more