Quick question on setting fs.s3a.endpoint

[bfleming-ciena]

I am told that if I can set fs.s3a.endpoint=s3.ca-central-1.amazonaws.com I can support the S3 V4 signature in the new regions so saveAsText file won't fail.

However, could someone please chime in as to where I set this value in spark-ec2 and what I need to do for it to take effect?

Would be a big help! Thanks.

[nchammas]

These don't seem like questions about the spark-ec2 tool itself. I think this question and #102 probably belong better on the Spark Users mailing list or on Stack Overflow.

Issues here should be restricted to issues with the spark-ec2 tool itself, not with Spark in general or with generic Spark interactions with AWS.

[bfleming-ciena]

Add this to spark-defaults.conf on the master.

Support Canada Region which uses the V4 S3 signature. Otherwise you will see "bad request" messages from saveAsTextFile

spark.driver.extraJavaOptions -Dcom.amazonaws.services.s3.enableV4 -Dcom.amazonaws.services.s3.enforceV4
spark.executor.extraJavaOptions -Dcom.amazonaws.services.s3.enableV4 -Dcom.amazonaws.services.s3.enforceV4