Add broken remote link checker (com-lihaoyi#3738)

Will probably need to fix a lot of links before this can land

contrib/bloop/readme.adoc

@@ -46,5 +46,6 @@ located inside a project workspace.
 
 == Note regarding current mill support in bloop
 
-The mill-bloop integration currently present in the https://github.com/scalacenter/bloop/blob/master/integrations/mill-bloop/src/main/scala/bloop/integrations/mill/MillBloop.scala#L10[bloop codebase]
+The mill-bloop integration currently present in the
+https://github.com/scalacenter/bloop[bloop codebase]
 will be deprecated in favour of this implementation.

contrib/testng/readme.adoc

@@ -2,7 +2,7 @@
 :page-aliases: TestNG_TestFramework.adoc
 
 
-Provides support for https://testng.org/doc/index.html[TestNG].
+Provides support for https://testng.org[TestNG].
 
 To use TestNG as test framework, you need to add it to the `TestModule.testFramework` property.
 

docs/modules/ROOT/pages/depth/design-principles.adoc

@@ -182,7 +182,7 @@ that would make a build tool hard to understand.
 Before you continue, take a moment to think: how would you answer to each of
 those questions using an existing build tool you are familiar with? Different
 tools like http://www.scala-sbt.org/[SBT],
-https://fake.build/legacy-index.html[Fake], https://gradle.org/[Gradle] or
+https://fake.build[Fake], https://gradle.org/[Gradle] or
 https://gruntjs.com/[Grunt] have very different answers.
 
 Mill aims to provide the answer to these questions using as few, as familiar

docs/modules/ROOT/pages/extending/thirdparty-plugins.adoc

@@ -571,8 +571,8 @@ bash> mill site.jbakeServe
 
 == JBuildInfo
 
-This is a https://www.lihaoyi.com/mill/[mill] module similar to
-https://www.lihaoyi.com/mill/page/contrib-modules.html#buildinfo[BuildInfo]
+This is a Mill module similar to
+xref:contrib/buildinfo.adoc[BuildInfo]
 but for Java.
 It will generate a Java class containing information from your build.
 

docs/modules/ROOT/partials/Installation_IDE_Support.adoc

@@ -274,7 +274,7 @@ CAUTION: Some of the installations via package managers install a fixed version
 
 === OS X
 
-Installation via https://github.com/Homebrew/homebrew-core/blob/master/Formula/mill.rb[homebrew]:
+Installation via https://github.com/Homebrew/homebrew-core/blob/master/Formula/m/mill.rb[homebrew]:
 
 [source,sh]
 ----

docs/package.mill

@@ -368,14 +368,72 @@ object `package` extends RootModule {
       )
     }
   }
+
+  def brokenRemoteLinks: T[Map[os.Path, Seq[(String, String, Int)]]] = Task{
+    val allLinks = allLinksAndAnchors()
+      .flatMap { case (path, remoteLinks, localLinks, ids) => remoteLinks }
+      .map(_._2)
+      .filter{l =>
+        // ignore example links since those are expected to be unresolved until
+        // a stable version is published and artifacts are uploaded to github
+        !l.contains("/example/") &&
+          !l.contains("/releases/download/") &&
+          // Ignore internal repo links in the changelog because there are a lot
+          // of them and they're not very interesting to check and verify.
+          !l.contains("https://github.com/com-lihaoyi/mill/pull/") &&
+          !l.contains("https://github.com/com-lihaoyi/mill/milestone/") &&
+          !l.contains("https://github.com/com-lihaoyi/mill/compare/") &&
+          // Link meant for API configuration, not for clicking
+          !l.contains("https://s01.oss.sonatype.org/service/local") &&
+          // SOmehow this server doesn't respond properly to HEAD requests even though GET works
+          !l.contains("https://marketplace.visualstudio.com/items")
+      }
+      .toSet
+
+    // Try to fetch all the links serially. It isn't worth trying to parallelize it
+    // because if we go too fast the remote websites tend to rate limit us anyway
+    val linksToStatusCodes = allLinks.toSeq.zipWithIndex
+      .map{ case (link, i) =>
+        val key = s"$i/${allLinks.size}"
+        println(s"Checking link $link $key")
+        val start = System.currentTimeMillis()
+        val res = requests.head(link, check = false).statusCode
+        val duration = System.currentTimeMillis() - start
+        val remaining = 1000 - duration
+        if (remaining > 0) Thread.sleep(remaining) // try to avoid being rate limited
+        (link, res)
+      }
+      .toMap
+
+    allLinksAndAnchors()
+      .map{case (path, remoteLinks, localLinks, ids) =>
+        (
+          path,
+          remoteLinks.collect{
+            case (e, l)
+              if allLinks.contains(l)
+              && !linksToStatusCodes(l).toString.startsWith("2")  =>
+                (e, l, linksToStatusCodes(l))
+          }
+        )
+      }
+      .filter(_._2.nonEmpty)
+      .toMap
+  }
+
+
   def checkBrokenLinks() = Task.Command{
-    if (brokenLinks().nonEmpty){
-      throw new Exception("Broken Links: " + upickle.default.write(brokenLinks(), indent = 2))
+    if (brokenLocalLinks().nonEmpty){
+      throw new Exception("Broken Local Links: " + upickle.default.write(brokenLocalLinks(), indent = 2))
+    }
+    if (brokenRemoteLinks().nonEmpty){
+      throw new Exception("Broken Rmote Links: " + upickle.default.write(brokenRemoteLinks(), indent = 2))
     }
   }
-  def brokenLinks: T[Map[os.Path, Seq[(String, String)]]] = Task{
+
+  def brokenLocalLinks: T[Map[os.Path, Seq[(String, String)]]] = Task{
     val allLinksAndAnchors0 = allLinksAndAnchors()
-    val pathsToIds = allLinksAndAnchors()
+    val pathsToIds = allLinksAndAnchors0
       .map{case (path, remoteLinks, localLinks, ids) => (path, ids)}
       .toMap
 

example/javalib/linting/1-error-prone/build.mill

@@ -51,7 +51,7 @@ object `package` extends RootModule with JavaModule with ErrorProneModule {
 //
 // `def errorProneVersion: T[String]`::
 // The `error-prone` version to use. Defaults to [[BuildInfo.errorProneVersion]], the version used to build and test the module.
-// Find the latest at https://mvnrepository.com/artifact/com.google.errorprone/error_prone_core[mvnrepository.com]
+// Find the list of versions and changlog at https://github.com/google/error-prone/releases
 //
 // `def errorProneOptions: T[Seq[String]]`::
 //  Options directly given to the `error-prone` processor.