aws emr interview questions

What is AWS EMR | Introduction to Amazon EMR | Data Processing with AWS EMR | AWS Training | Edureka

Impala executes SQL queries using a massively parallel processing (MPP) engine, while Hive executes SQL queries using MapReduce. Impala avoids Hive’s overhead from creating MapReduce jobs, giving it faster query times than Hive. However, Impala uses significant memory resources and the cluster’s available memory places a constraint on how much memory any query can consume. Hive is not limited in the same way, and can successfully process larger data sets with the same hardware. Generally, you should use Impala for fast, interactive queries, while Hive is better for ETL workloads on large datasets. Impala is built for speed and is great for ad hoc investigation, but requires a significant amount of memory to execute expensive queries or process very large datasets. Because of these limitations, Hive is recommended for workloads where speed is not as crucial as completion. Click here to view some performance benchmarks between Impala and Hive.

We recommend that new customers use Amazon EMR Studio, not EMR Notebooks. EMR Notebooks provide a managed environment, based on Jupyter Notebook, that allows data scientists, analysts, and developers to prepare and visualize data, collaborate with peers, build applications, and perform interactive analysis using EMR clusters. Although we recommend that new customers use EMR Studio, EMR Notebooks is supported for compatibility.

Customers can create a base , add their corporate standard libraries, and then store it in Amazon Elastic Container Registry (Amazon ECR). Other customers can customize the to include their application specific dependencies. The resulting immutable can be vulnerability scanned, deployed to test and production environments. Examples of dependencies you can add include Java SDK, Python, or R libraries, you can add them to the directly, just as with other containerized applications.

The Hadoop MapReduce framework is a batch processing system. As such, it does not support continuous queries. However there is an emerging set of Hadoop ecosystem frameworks like Twitter Storm and Spark Streaming that enable to developers build applications for continuous stream processing. A Storm connector for Kinesis is available at on GitHub here and you can find a tutorial explaining how to setup Spark Streaming on EMR and run continuous queries here.

a/ Accessing multiple filesystems. By default a Pig job can only access one remote file system, be it an HDFS store or S3 bucket, for input, output and temporary data. EMR has extended Pig so that any job can access as many file systems as it wishes. An advantage of this is that temporary intra-job data is always stored on the local HDFS, leading to improved performance.

1) Explain what AWS is?AWS stands for Amazon Web Service; it is a collection of remote computing services also known as a cloud computing platform.  This new realm of cloud computing is also known as IaaS or Infrastructure as a Service.

The key components of AWS are

  • Route 53: A DNS web service
  • Simple E-mail Service: It allows sending e-mail using RESTFUL API call or via regular SMTP
  • Identity and Access Management: It provides enhanced security and identity management for your AWS account
  • Simple Storage Device or (S3): It is a storage device and the most widely used AWS service
  • Elastic Compute Cloud (EC2): It provides on-demand computing resources for hosting applications. It is handy in case of unpredictable workloads
  • Elastic Block Store (EBS): It offers persistent storage volumes that attach to EC2 to allow you to persist data past the lifespan of a single Amazon EC2 instance
  • CloudWatch: To monitor AWS resources, It allows administrators to view and collect keys. Also, one can set a notification alarm in case of trouble.
  • 13. How does Amazon Route 53 provide high availability and low latency?

    Amazon Route 53 uses the following to provide high availability and low latency:

  • Globally Distributed Servers – Amazon is a global service and consequently has DNS Servers globally. Any customer creating a query from any part of the world gets to reach a DNS Server local to them that provides low latency.
  • Dependency – Route 53 provides a high level of dependability required by critical applications.
  • Optimal Locations – Route 53 serves the requests from the nearest data center to the client sending the request. AWS has data-centers across the world. The data can be cached on different data-centers located in different regions of the world depending on the requirements and the configuration chosen. Route 53 enables any server in any data-center which has the required data to respond. This way, it enables the nearest server to serve the client request, thus reducing the time taken to serve.
  • As can be seen in the above , the requests coming from a user in India are served from the Singapore Server, while the requests coming from a user in the US are routed to Oregon region.

    FAQ

    What is AWS EMR used for?

    Amazon EMR (previously called Amazon Elastic MapReduce) is a managed cluster platform that simplifies running big data frameworks, such as Apache Hadoop and Apache Spark , on AWS to process and analyze vast amounts of data.

    What is Ami in AWS interview questions?

    Basic AWS Interview Questions
    1. Define and explain the three basic types of cloud services and the AWS products that are built based on them? …
    2. What is the relation between the Availability Zone and Region? …
    3. What is auto-scaling? …
    4. What is geo-targeting in CloudFront? …
    5. What are the steps involved in a CloudFormation Solution?

    What is AWS EMR step?

    AMI stands for Amazon Machine Image. It’s a template that provides the information (an operating system, an application server, and applications) required to launch an instance, which is a copy of the AMI running as a virtual server in the cloud.

    Related Posts

    Leave a Reply

    Your email address will not be published.