Twitter jars not getting downloaded #1051

piyuverma · 2019-11-06T12:10:38Z

Build failure while fetching twitter jars.

HenryCaiHaiying · 2019-11-06T19:48:32Z

On Wed, Nov 6, 2019 at 4:10 AM piyuverma ***@***.***> wrote: Build failure while fetching twitter jars. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1051?email_source=notifications&email_token=ABYJP72DEOSI644KFJ56G6DQSKX4DA5CNFSM4JJT7X4KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HXGUPLQ>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYJP72NOA36PVOWBHSAWULQSKX4DANCNFSM4JJT7X4A> .

HenryCaiHaiying · 2019-11-06T22:25:09Z

It seems the folks at GCS mirror is unwilling to sync the jars from Twitter maven. Also opened a support ticket to Travis to see whether they can stop mirroring all repos to GCS.

…

On Wed, Nov 6, 2019 at 11:48 AM Henry Cai ***@***.***> wrote: Sent a request to ***@***.*** On Wed, Nov 6, 2019 at 4:10 AM piyuverma ***@***.***> wrote: > Build failure while fetching twitter jars. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#1051?email_source=notifications&email_token=ABYJP72DEOSI644KFJ56G6DQSKX4DA5CNFSM4JJT7X4KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HXGUPLQ>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABYJP72NOA36PVOWBHSAWULQSKX4DANCNFSM4JJT7X4A> > . >

HenryCaiHaiying · 2019-11-06T22:26:39Z

If Travis folks is unwilling to revert their change, we probably have to use our own mirrors to disable GCS mirror as suggested here: https://travis-ci.community/t/disable-google-maven-central/5301

…

On Wed, Nov 6, 2019 at 2:24 PM Henry Cai ***@***.***> wrote: It seems the folks at GCS mirror is unwilling to sync the jars from Twitter maven. Also opened a support ticket to Travis to see whether they can stop mirroring all repos to GCS. On Wed, Nov 6, 2019 at 11:48 AM Henry Cai ***@***.***> wrote: > Sent a request to ***@***.*** > > On Wed, Nov 6, 2019 at 4:10 AM piyuverma ***@***.***> > wrote: > >> Build failure while fetching twitter jars. >> >> — >> You are receiving this because you are subscribed to this thread. >> Reply to this email directly, view it on GitHub >> <#1051?email_source=notifications&email_token=ABYJP72DEOSI644KFJ56G6DQSKX4DA5CNFSM4JJT7X4KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HXGUPLQ>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/ABYJP72NOA36PVOWBHSAWULQSKX4DANCNFSM4JJT7X4A> >> . >> >

sreeram26 · 2019-11-08T23:17:39Z

Do we have any updates on this?

HenryCaiHaiying · 2019-11-09T00:37:53Z

No updates on Travis side. If you have time, you can prepare a PR to add a setting.xml which defines another mirror override to override the twitter maven repo back to twitter (instead of Google mirror)

…

On Fri, Nov 8, 2019 at 3:17 PM Sreeram Ramji ***@***.***> wrote: Do we have any updates on this? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1051?email_source=notifications&email_token=ABYJP72WM4EDXP44OZEEKULQSXXRNA5CNFSM4JJT7X4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDTUOUI#issuecomment-552027985>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYJP7ZJZEC4GFKM6AHSJ4DQSXXRNANCNFSM4JJT7X4A> .

sreeram26 · 2019-11-09T01:36:56Z

Sorry I don't quiet follow.
I see https://maven.twttr.com/com/twitter/maven-finagle-thrift-plugin/0.0.9/ also missing the corresponding jar?
Same case with most other repositories in twitter. What would be the exact change in settings.xml which would solve this?

HenryCaiHaiying · 2019-11-09T17:29:51Z

Take a look at how they specify repo mirror in settings.xml (essentially cancelled off google's mirror), we need this for twitter's maven repo mirror. ksclarke <https://travis-ci.community/u/ksclarke> 2 <https://travis-ci.community/t/disable-google-maven-central/5301#> Oct 7 <https://travis-ci.community/t/disable-google-maven-central/5301/2> I want this too. Google’s mirror of Maven Central is taking too long to sync with the upstream. *Update:* To get around the Google mirror, you can set the real Maven central as a mirror in a settings.xml file and pass that to the mvn command with -s. For an example: https://github.com/ksclarke/vertx-super-s3/blob/master/src/test/resources/settings.xml#L21 5 <https://github.com/ksclarke/vertx-super-s3/blob/master/src/test/resources/settings.xml#L21> and https://github.com/ksclarke/vertx-super-s3/blob/master/.travis.yml#L17 6 <https://github.com/ksclarke/vertx-super-s3/blob/master/.travis.yml#L17>

…

On Fri, Nov 8, 2019 at 5:36 PM Sreeram Ramji ***@***.***> wrote: Sorry I don't quiet follow. I see https://maven.twttr.com/com/twitter/maven-finagle-thrift-plugin/0.0.9/ also missing the corresponding jar? Same case with most other repositories. What would be the exact change in settings.xml which would solve this? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1051?email_source=notifications&email_token=ABYJP7YP34OSCAUBDI7WOQLQSYH3XA5CNFSM4JJT7X4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDT2NSY#issuecomment-552052427>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYJP76FEQZHB5LKCZXZ6LTQSYH3XANCNFSM4JJT7X4A> .

sreeram26 · 2019-11-11T17:53:40Z

I tried that

:secor_local sreeram.ramji$ cat settings.xml
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <mirrors>
    <mirror>
      <id>maven-central</id>
      <name>Maven Central</name>
      <url>http://repo1.maven.org/maven2</url>
      <mirrorOf>twitter</mirrorOf>
    </mirror>
  </mirrors>
</settings>
secor_local sreeram.ramji$ mvn -U -s settings.xml package
[INFO] Scanning for projects...
[ERROR] Failed to execute goal on project secor: Could not resolve dependencies for project com.pinterest:secor:jar:0.29-SNAPSHOT: The following artifacts could not be resolved: com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16, com.twitter.common.zookeeper:lock:jar:0.0.7: Could not find artifact com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16 in maven-central (http://repo1.maven.org/maven2) -> [Help 1]

As these are not present in maven central's repo too see http://repo1.maven.org/maven2/com/hadoop/compression/hadoop-gpl-compression/0.1.0/

HenryCaiHaiying · 2019-11-11T18:06:26Z

Those jars are only in twitter's maven repo How does your PR look like?

…

On Mon, Nov 11, 2019 at 9:53 AM Sreeram Ramji ***@***.***> wrote: I tried that [ERROR] Failed to execute goal on project secor: Could not resolve dependencies for project com.pinterest:secor:jar:0.29-SNAPSHOT: The following artifacts could not be resolved: com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16, com.twitter.common.zookeeper:lock:jar:0.0.7: Could not find artifact com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16 in maven-central (http://repo1.maven.org/maven2) -> [Help 1] As these are not present in maven central's repo too see http://repo1.maven.org/maven2/com/hadoop/compression/hadoop-gpl-compression/0.1.0/ — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1051?email_source=notifications&email_token=ABYJP7ZHRCN5MC3ASYBWZ6TQTGL2PA5CNFSM4JJT7X4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDXS7SQ#issuecomment-552546250>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYJP73H2U43IQCID2AZPPTQTGL2PANCNFSM4JJT7X4A> .

sreeram26 · 2019-11-11T18:29:53Z

Yes exactly, and the twitter's maven repo seems to miss the corresponding jars.
One example is
https://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.20/

Raised twitter/hadoop-lzo#140, but no updates so far. Similarly for all of the other jars available only in twitter's maven repo.

HenryCaiHaiying · 2019-11-11T18:56:15Z

It looks like 0.4.16 hadoop-lzo are still there: https://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.16/

…

On Mon, Nov 11, 2019 at 10:29 AM Sreeram Ramji ***@***.***> wrote: Yes exactly, and the twitter's maven repo seems to miss the corresponding jars. One example is https://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.20/ Raised twitter/hadoop-lzo#140 <twitter/hadoop-lzo#140>, but no updates so far. Similarly for all of the other jars available only in twitter's maven repo. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1051?email_source=notifications&email_token=ABYJP7ZJBKVPZF7W7SBQRK3QTGQCLA5CNFSM4JJT7X4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDXWEYA#issuecomment-552559200>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYJP75Y3LYH5YUJHNMZSMTQTGQCLANCNFSM4JJT7X4A> .

piyuverma · 2019-11-11T19:02:12Z

No, for 0.4.16 hadoop-lzo also, just the .pom files and md5 and sha1 hash files are there but not the actual .jar file.

msilvestre · 2019-11-12T09:03:31Z

Is there any workaround for this issue?

guillem · 2019-11-12T10:42:44Z

Do we know what's going on with Twitter? Is this a mistake or they've removed the files for good?
Since they're still publishing the hashes, can someone who still has the files at least share them please, so we can manually make the compilation work? Thanks.

piyuverma · 2019-11-12T13:00:23Z

I have all the jars except for the finagle plugin, i tried commenting out the plugin in pom and with installing all the jars locally in order to get away with the compile time build error. The build succeeded, but got NoClassDefFoundError at runtime for one of the Class InetSocketAddressHelper.java present in net-util.jar. However, that class was present at compile time.
Please download jars here https://drive.google.com/open?id=1WpVlxvJZ5Ip-o9zIg3Q8QsIzormrdzTy

msilvestre · 2019-11-14T16:03:31Z

Any news regarding this issue?

HenryCaiHaiying · 2019-11-27T01:45:17Z

Got some feedback from https://gitter.im/twitter/finagle, they asked us to upgrade finagle-maven-plugin to scrooge-maven-plugin

sreeram26 · 2019-11-27T02:11:40Z

There are other dependencies from twitter like https://maven.twttr.com/com/twitter/common/zookeeper/lock/ what are your plans for that?

Sreerams-MacBook-Pro-2:secor sreeram.ramji$ mvn dependency:tree | grep twitter
[INFO] |  \- com.twitter.elephantbird:elephant-bird-core:jar:4.4:compile
[INFO] |     +- com.twitter.elephantbird:elephant-bird-hadoop-compat:jar:4.4:compile
[INFO] |  +- com.twitter.elephantbird:elephant-bird-pig:jar:4.4:compile
[INFO] +- com.twitter.common.zookeeper:lock:jar:0.0.7:compile
[INFO] |  +- com.twitter.common.zookeeper:client:jar:0.0.7:compile
[INFO] |  |  +- com.twitter.common:net-util:jar:0.0.29:compile
[INFO] |  |  |  \- com.twitter.common:collections:jar:0.0.34:compile
[INFO] |  |  \- com.twitter.common:quantity:jar:0.0.30:compile
[INFO] |  +- com.twitter.common:base:jar:0.0.38:compile
[INFO] |  |  \- com.twitter.common:util-system-mocks:jar:0.0.26:compile
[INFO] \- com.twitter:ostrich_2.10:jar:9.18.0:compile
[INFO]    +- com.twitter:util-core_2.10:jar:6.34.0:compile
[INFO]    |  +- com.twitter:util-function_2.10:jar:6.34.0:compile
[INFO]    |  \- com.twitter:jsr166e:jar:1.0.0:compile
[INFO]    +- com.twitter:util-eval_2.10:jar:6.34.0:compile
[INFO]    +- com.twitter:util-logging_2.10:jar:6.34.0:compile
[INFO]    |  +- com.twitter:util-app_2.10:jar:6.34.0:compile
[INFO]    |  |  \- com.twitter:util-registry_2.10:jar:6.34.0:compile
[INFO]    |  \- com.twitter:util-stats_2.10:jar:6.34.0:compile
[INFO]    |     \- com.twitter:util-lint_2.10:jar:6.34.0:compile
[INFO]    +- com.twitter:util-jvm_2.10:jar:6.34.0:compile
[INFO]    \- com.twitter:scala-json_2.10:jar:3.0.2:compile

I had to manually add

$ ls
base-0.0.38.jar			client-0.0.7.jar		collections-0.0.34.jar		hadoop-lzo-0.4.16.jar		lock-0.0.7.jar			net-util-0.0.29.jar		quantity-0.0.30.jar		util-system-mocks-0.0.26.jar

to make the build pass

HenryCaiHaiying · 2019-11-27T02:22:33Z

One at a time, I am upgrading maven-plugin first, to see if this PR can pass through build:
#1087

HenryCaiHaiying · 2019-11-27T02:24:49Z

Looks like we passed maven plugin block, now it's stuck at:

The following artifacts could not be resolved: com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16, com.twitter.common.zookeeper:lock:jar:0.0.7, com.twitter.common.zookeeper:client:jar:0.0.7, com.twitter.common:net-util:jar:0.0.29, com.twitter.common:collections:jar:0.0.34, com.twitter.common:quantity:jar:0.0.30, com.twitter.common:base:jar:0.0.38, com.twitter.common:util-system-mocks:jar:0.0.26:

HenryCaiHaiying · 2019-11-27T21:00:32Z

Almost all twitter.common jars are pulled in because of com.twitter.common.zookeeper:lock:jar:0.0.7, twitter people asked us to upgrade to finagle-serverset

HenryCaiHaiying · 2019-11-27T21:47:37Z

Actually apache.curator package is probably more appropriate for zookeeper locks

HenryCaiHaiying · 2019-11-27T22:32:46Z

Made another commit to use apache.curator: #1087

Now we are only missing one jar: com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16,

And this is pulled from this path:

[INFO] +- org.apache.parquet:parquet-protobuf:jar:1.9.0:compile
[INFO] | - com.twitter.elephantbird:elephant-bird-core:jar:4.4:compile
[INFO] | +- com.twitter.elephantbird:elephant-bird-hadoop-compat:jar:4.4:compile
[INFO] | +- com.googlecode.json-simple:json-simple:jar:1.1:compile
[INFO] | - com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16:compile

HenryCaiHaiying · 2019-11-27T23:30:58Z

Made one more fix in pom.xml to get rid of the dependency on hadoop-lzo, let's see how far the build is going this time

HenryCaiHaiying · 2019-12-01T08:22:03Z

Make another fix: #1090

Looks like the build is working now.

guillem · 2019-12-02T14:50:44Z

Thank you guys :)

HenryCaiHaiying mentioned this issue Nov 8, 2019

Build failed - Plugin com.twitter:maven-finagle-thrift-plugin:0.0.9 or one of its dependencies could not be resolved #282

Closed

HenryCaiHaiying closed this as completed Dec 1, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Twitter jars not getting downloaded #1051

Twitter jars not getting downloaded #1051

piyuverma commented Nov 6, 2019

HenryCaiHaiying commented Nov 6, 2019 via email

HenryCaiHaiying commented Nov 6, 2019 via email

HenryCaiHaiying commented Nov 6, 2019 via email

sreeram26 commented Nov 8, 2019

HenryCaiHaiying commented Nov 9, 2019 via email

sreeram26 commented Nov 9, 2019 •

edited

Loading

HenryCaiHaiying commented Nov 9, 2019 via email

sreeram26 commented Nov 11, 2019 •

edited

Loading

HenryCaiHaiying commented Nov 11, 2019 via email

sreeram26 commented Nov 11, 2019

HenryCaiHaiying commented Nov 11, 2019 via email

piyuverma commented Nov 11, 2019

msilvestre commented Nov 12, 2019

guillem commented Nov 12, 2019

piyuverma commented Nov 12, 2019 •

edited

Loading

msilvestre commented Nov 14, 2019

HenryCaiHaiying commented Nov 27, 2019 •

edited

Loading

sreeram26 commented Nov 27, 2019 •

edited

Loading

HenryCaiHaiying commented Nov 27, 2019

HenryCaiHaiying commented Nov 27, 2019

HenryCaiHaiying commented Nov 27, 2019

HenryCaiHaiying commented Nov 27, 2019

HenryCaiHaiying commented Nov 27, 2019

HenryCaiHaiying commented Nov 27, 2019

HenryCaiHaiying commented Dec 1, 2019

guillem commented Dec 2, 2019

Twitter jars not getting downloaded #1051

Twitter jars not getting downloaded #1051

Comments

piyuverma commented Nov 6, 2019

HenryCaiHaiying commented Nov 6, 2019 via email

HenryCaiHaiying commented Nov 6, 2019 via email

HenryCaiHaiying commented Nov 6, 2019 via email

sreeram26 commented Nov 8, 2019

HenryCaiHaiying commented Nov 9, 2019 via email

sreeram26 commented Nov 9, 2019 • edited Loading

HenryCaiHaiying commented Nov 9, 2019 via email

sreeram26 commented Nov 11, 2019 • edited Loading

HenryCaiHaiying commented Nov 11, 2019 via email

sreeram26 commented Nov 11, 2019

HenryCaiHaiying commented Nov 11, 2019 via email

piyuverma commented Nov 11, 2019

msilvestre commented Nov 12, 2019

guillem commented Nov 12, 2019

piyuverma commented Nov 12, 2019 • edited Loading

msilvestre commented Nov 14, 2019

HenryCaiHaiying commented Nov 27, 2019 • edited Loading

sreeram26 commented Nov 27, 2019 • edited Loading

HenryCaiHaiying commented Nov 27, 2019

HenryCaiHaiying commented Nov 27, 2019

HenryCaiHaiying commented Nov 27, 2019

HenryCaiHaiying commented Nov 27, 2019

HenryCaiHaiying commented Nov 27, 2019

HenryCaiHaiying commented Nov 27, 2019

HenryCaiHaiying commented Dec 1, 2019

guillem commented Dec 2, 2019

sreeram26 commented Nov 9, 2019 •

edited

Loading

sreeram26 commented Nov 11, 2019 •

edited

Loading

piyuverma commented Nov 12, 2019 •

edited

Loading

HenryCaiHaiying commented Nov 27, 2019 •

edited

Loading

sreeram26 commented Nov 27, 2019 •

edited

Loading