installing packages in AWS EMR

Question

I'm trying to install Google Tink in AWS EMR 5.28.0 without much luck. It looks like the AWS EMR image is rather strange in nature. Any ideas?

sc.install_pypi_package("tink")

error: Could not find bazel executable. Please install bazel to compile the Tink Python package.

 sudo yum install -y bazel3

Requires: java-11-openjdk-devel

sudo yum install java-11-openjdk-devel

No package java-11-openjdk-devel available.

 sudo yum install amazon-linux-extras

No package amazon-linux-extras available.

You are new to Linux? Start with the [documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/amazon-linux-ami-basics.html#package-repository). — Michael Hampton, Aug 13 '21 at 21:04
@MichaelHampton I have enabled sudo yum-config-manager --enable epel. No change, I would have thought jdk is a pretty common package. — Koenig Lear, Aug 16 '21 at 10:43
@MichaelHampton yes how do you install them? I can't see anything in the aws doc — Koenig Lear, Aug 16 '21 at 10:48
Now you know about yum-config-manager. You can use that to inspect the state of the system and make any necessary changes. — Michael Hampton, Aug 16 '21 at 10:51
I am not here to read documentation for you. You should have already read about yum-config-manager and learned what you can do with it. — Michael Hampton, Aug 16 '21 at 11:01

score 0 · Accepted Answer · answered Aug 16 '21 at 12:45

As it turns out the Linux AMI (The base for AWS EMR 5.28.0) is a pain to work with. None of the available repositories have Java, Bezel, much less Tinker readily available (compare to Macos, Ubuntu, or even ECR Python image where it's a breeze)

To get Java 11jdk you need to go to Oracle and look for Linux x64 RPM Package. Unfortunately you need to sign in to their site so wget/curl is not possible.

To install you need to do:

 sudo yum localinstall jdk-11.0.12_linux-x64_bin.rpm

For Bazel you need to get the package from Github https://github.com/bazelbuild/. Pick the latest and install

./bazel-4.1.0-installer-linux-x86_64.sh

Then pip works fine

sc.install_pypi_package("tink")

Output:

Successfully built tink
Installing collected packages: protobuf, absl-py, tink
Successfully installed absl-py-0.13.0 protobuf-3.17.3 tink-1.6.1

installing packages in AWS EMR

1 Answers1