From 8145577453c672f6940a64dd075a700bbad1d915 Mon Sep 17 00:00:00 2001 From: Mike Barry Date: Wed, 25 Oct 2023 21:16:27 -0400 Subject: [PATCH 1/4] java 21 --- CONTRIBUTING.md | 8 ++++---- PLANET.md | 8 ++++---- README.md | 6 +++--- planetiler-custommap/README.md | 12 ++++++++---- planetiler-dist/pom.xml | 2 +- planetiler-examples/README.md | 2 +- planetiler-examples/standalone.pom.xml | 4 ++-- planetiler-openmaptiles | 2 +- pom.xml | 20 +++++++++++++++----- 9 files changed, 39 insertions(+), 25 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a55bb979f7..0bd8b1a20e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -11,12 +11,12 @@ Pull requests are welcome! Any pull request should: To set up your local development environment: - Fork the repo [setup submodules](README.md#git-submodules) -- Install Java 17 or later. You can download Java manually from [Adoptium](https://adoptium.net/installation.html) or +- Install Java 21 or later. You can download Java manually from [Adoptium](https://adoptium.net/installation.html) or use: - [Windows installer](https://adoptium.net/installation.html#windows-msi) - [macOS installer](https://adoptium.net/installation.html#macos-pkg) (or `brew install --cask temurin`, - or `port install openjdk17-temurin`) - - [Linux installer](https://adoptium.net/installation/linux/) (or `apt-get install openjdk-17-jdk`) + or `port install openjdk21-temurin`) + - [Linux installer](https://adoptium.net/installation/linux/) (or `apt-get install openjdk-21-jdk`) - Build and run the tests ([mvnw](https://github.com/takari/maven-wrapper) automatically downloads maven the first time you run it): - on mac/linux: `./mvnw clean test` @@ -54,7 +54,7 @@ Troubleshooting: - If any java source files show "Cannot resolve symbol..." errors for Planetiler classes, you might need to select: `File -> Invalidate Caches... -> Just Restart`. -- If you see a "Project JDK is not defined" error, then choose `Setup SDK` and point IntelliJ at the Java 17 or later +- If you see a "Project JDK is not defined" error, then choose `Setup SDK` and point IntelliJ at the Java 21 or later installed on your system ### Visual Studio Code diff --git a/PLANET.md b/PLANET.md index 6c8bd268c4..b0fa28e662 100644 --- a/PLANET.md +++ b/PLANET.md @@ -3,9 +3,9 @@ To generate a map of the world using the built-in [OpenMapTiles profile](https://github.com/openmaptiles/planetiler-openmaptiles), you will need a machine with -Java 17 or later installed and at least 10x as much disk space and at least 0.5x as much RAM as the `planet.osm.pbf` +Java 21 or later installed and at least 10x as much disk space and at least 0.5x as much RAM as the `planet.osm.pbf` file you start from. All testing has been done using Digital Ocean droplets with dedicated -vCPUs ([referral link](https://m.do.co/c/a947e99aab25)) and OpenJDK 17 installed through `apt`. Planetiler splits work +vCPUs ([referral link](https://m.do.co/c/a947e99aab25)) and OpenJDK 21 installed through `apt`. Planetiler splits work among available CPUs so the more you have, the less time it takes. ### 1) Choose the Data Source @@ -84,10 +84,10 @@ To generate the tiles shown on https://onthegomap.github.io/planetiler-demo/ I u S3 snapshot, then ran Planetiler on a Digital Ocean Memory-Optimized droplet with 16 CPUs, 128GB RAM, and 1.17TB disk running Ubuntu 21.04 x64 in the nyc3 location. -First, I installed java 17 jre and screen: +First, I installed java 21 jre and screen: ```bash -apt-get update && apt-get install -y openjdk-17-jre-headless screen +apt-get update && apt-get install -y openjdk-21-jre-headless screen ``` Then I added a script `runworld.sh` to run with 100GB of RAM: diff --git a/README.md b/README.md index 25bfa2ec78..f5e82477e0 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ the [OpenStreetMap Americana Project](https://github.com/ZeLonewolf/openstreetma To generate a map of an area using the [OpenMapTiles profile](https://github.com/openmaptiles/planetiler-openmaptiles), you will need: -- Java 17+ (see [CONTRIBUTING.md](CONTRIBUTING.md)) or [Docker](https://docs.docker.com/get-docker/) +- Java 21+ (see [CONTRIBUTING.md](CONTRIBUTING.md)) or [Docker](https://docs.docker.com/get-docker/) - at least 1GB of free disk space plus 5-10x the size of the `.osm.pbf` file - at least 0.5x as much free RAM as the input `.osm.pbf` file size @@ -162,7 +162,7 @@ consider [contributing](#contributing) your change back for others to use! Some example runtimes for the OpenMapTiles profile (excluding downloading resources): -| Input | Version | Machine | Time | mbtiles size | Logs | +| Input | Version | Machine | Time | mbtiles size | Logs | |-------------------------------------------------------------------------------------------------------------------------------------------|---------|---------------------------------|---------------------------------|--------------|------------------------------------------------------------------------------------------------------------------------| | s3://osm-pds/2022/planet-220530.osm.pbf (69GB) | 0.5.0 | c2d-standard-112 (112cpu/448GB) | 37m cpu:48h5m gc:3m45s avg:76.9 | 79GB | [logs](planet-logs/v0.5.0-planet-c2d-standard-112.txt) | | s3://osm-pds/2022/planet-220530.osm.pbf (69GB) | 0.5.0 | c6gd.16xlarge (64cpu/128GB) | 53m cpu:41h58m avg:47.1 | 79GB | [logs](planet-logs/v0.5.0-planet-c6gd-128gb.txt), [VisualVM Profile](planet-logs/v0.5.0-planet-c6gd-128gb.nps) | @@ -173,7 +173,7 @@ Some example runtimes for the OpenMapTiles profile (excluding downloading resour Merging nearby buildings at z13 is very expensive, when run with `--building-merge-z13=false`: -| Input | Version | Machine | Time | mbtiles size | Logs | +| Input | Version | Machine | Time | mbtiles size | Logs | |------------------------------------------------|---------|----------------------------------------------------------|--------------------------|--------------|------------------------------------------------------------------------------------------------------------------------------------------------------------| | s3://osm-pds/2022/planet-220530.osm.pbf (69GB) | 0.5.0 | c2d-standard-112 (112cpu/448GB) | 26m cpu:27h47m avg:63.9 | 79GB | [logs](planet-logs/v0.5.0-planet-c2d-standard-112-no-z13-building-merge.txt) | | s3://osm-pds/2022/planet-220530.osm.pbf (69GB) | 0.5.0 | c6gd.16xlarge (64cpu/128GB) | 39m cpu:27h4m avg:42.1 | 79GB | [logs](planet-logs/v0.5.0-planet-c6gd-128gb-no-z13-building-merge.txt), [VisualVM Profile](planet-logs/v0.5.0-planet-c6gd-128gb-no-z13-building-merge.nps) | diff --git a/planetiler-custommap/README.md b/planetiler-custommap/README.md index 170b7ed04a..946df08a13 100644 --- a/planetiler-custommap/README.md +++ b/planetiler-custommap/README.md @@ -144,6 +144,7 @@ to regenerate: cat planetiler-custommap/planetiler.schema.json | jq -r '.properties.args.properties | to_entries[] | "- `" + .key + "` - " + .value.description' | pbcopy --> + - `threads` - Default number of threads to use. - `write_threads` - Default number of threads to use when writing temp features - `process_threads` - Default number of threads to use when processing input features @@ -292,15 +293,18 @@ Specific tile post processing operations for merging features may be defined: - `merge_line_strings` - Combines linestrings with the same set of attributes into a multilinestring where segments with touching endpoints are merged. -- `merge_polygons` - Combines polygons with the same set of attributes into a multipolygon where overlapping/touching polygons +- `merge_polygons` - Combines polygons with the same set of attributes into a multipolygon where overlapping/touching + polygons are combined into fewer polygons covering the same area. The follow attributes for `merge_line_strings` may be set: + - `min_length` - Minimum tile pixel length of features to emit, or 0 to emit all merged linestrings. - `tolerance` - After merging, simplify linestrings using this pixel tolerance, or -1 to skip simplification step. - `buffer` - Number of pixels outside the visible tile area to include detail for, or -1 to skip clipping step. The follow attribute for `merge_polygons` may be set: + - `min_area` - Minimum area in square tile pixels of polygons to emit. For example: @@ -476,7 +480,7 @@ nested, so each child context can also access the variables from its parent. >> ##### process feature context >> >> Context available when processing an input feature, for example testing whether to include it from `include_when`. ->> Available variables: +> > Available variables: >> >> - `feature.tags` - map with key/value tags from the input feature >> - `feature.id` - numeric ID of the input feature @@ -493,7 +497,7 @@ nested, so each child context can also access the variables from its parent. >>>> ##### configure attribute context >>>> >>>> Context available after the value of an attribute has been computed, for example: set min zoom to render an ->>>> attribute. Adds variables: +> > > > attribute. Adds variables: >>>> >>>> - `value` the value that was computed for this key @@ -534,7 +538,7 @@ in [PlanetilerStdLib](src/main/java/com/onthegomap/planetiler/custommap/expressi - `.replace(from, to, limit)` returns the input string with the first N occurrences of from replaced by to - `.replaceRegex(pattern, value)` replaces every occurrence of regular expression with value from the string it was called on using java's - built-in [replaceAll]() + built-in [replaceAll]() behavior - `.split(separator)` returns a list of strings split from the input by a separator - `.split(separator, limit)` splits the list into up to N parts diff --git a/planetiler-dist/pom.xml b/planetiler-dist/pom.xml index e147dd7f0d..af6ef5335b 100644 --- a/planetiler-dist/pom.xml +++ b/planetiler-dist/pom.xml @@ -54,7 +54,7 @@ false - eclipse-temurin:17-jre + eclipse-temurin:21-jre diff --git a/planetiler-examples/README.md b/planetiler-examples/README.md index c6e3922535..e7faea3949 100644 --- a/planetiler-examples/README.md +++ b/planetiler-examples/README.md @@ -4,7 +4,7 @@ This is a minimal example project that shows how to create custom maps with Plan Requirements: -- Java 17+ (see [CONTIRBUTING.md](../CONTRIBUTING.md)) +- Java 21+ (see [CONTIRBUTING.md](../CONTRIBUTING.md)) - on mac: `brew install --cask temurin` - [Maven](https://maven.apache.org/install.html) - on mac: `brew install maven` diff --git a/planetiler-examples/standalone.pom.xml b/planetiler-examples/standalone.pom.xml index d8fd00c1c3..f890777741 100644 --- a/planetiler-examples/standalone.pom.xml +++ b/planetiler-examples/standalone.pom.xml @@ -9,8 +9,8 @@ UTF-8 - 17 - 17 + 21 + 21 0.7-SNAPSHOT 5.10.0 diff --git a/planetiler-openmaptiles b/planetiler-openmaptiles index d4aaae894b..7dbbc5089e 160000 --- a/planetiler-openmaptiles +++ b/planetiler-openmaptiles @@ -1 +1 @@ -Subproject commit d4aaae894b02e6feb5d1ae042bf31c1bcfa73c49 +Subproject commit 7dbbc5089e8ca5607d9c3ecf1991d35b1de5c6cf diff --git a/pom.xml b/pom.xml index 153f422ad2..bd17f4d144 100644 --- a/pom.xml +++ b/pom.xml @@ -17,8 +17,8 @@ UTF-8 - 17 - 17 + 21 + 21 true 2.15.3 5.10.0 @@ -203,7 +203,7 @@ - + org.apache.maven.plugins maven-enforcer-plugin @@ -217,7 +217,7 @@ - 17 + 21 @@ -245,8 +245,18 @@ maven-deploy-plugin 3.1.1 + + org.apache.maven.plugins + maven-compiler-plugin + 3.11.0 + + + -proc:full + + + - + From b6aef02cc9d2b7e0600bf6bb46822574baed161e Mon Sep 17 00:00:00 2001 From: Mike Barry Date: Wed, 25 Oct 2023 21:22:31 -0400 Subject: [PATCH 2/4] update actions --- .github/workflows/maven.yml | 19 ++--- .github/workflows/performance.yml | 2 +- .github/workflows/release.yml | 120 ++++++++++++++-------------- .github/workflows/snapshot.yml | 68 ++++++++-------- .github/workflows/sonar.yml | 126 +++++++++++++++--------------- 5 files changed, 168 insertions(+), 167 deletions(-) diff --git a/.github/workflows/maven.yml b/.github/workflows/maven.yml index 1e416a65a5..fc2907da6a 100644 --- a/.github/workflows/maven.yml +++ b/.github/workflows/maven.yml @@ -18,10 +18,10 @@ jobs: - uses: actions/checkout@v4 with: submodules: true - - name: Set up JDK 17 + - name: Set up JDK 21 uses: actions/setup-java@v3 with: - java-version: 17 + java-version: 21 distribution: 'temurin' cache: 'maven' - name: Ensure code formatted with mvn spotless:apply @@ -34,13 +34,14 @@ jobs: fail-fast: false matrix: os: [ ubuntu-latest, macos-latest, windows-latest ] - jdk: [ 17 ] + jdk: [ 21 ] include: - os: ubuntu-latest - jdk: 17 + jdk: 21 args: "-DargLine='-Duser.language=fr -Duser.country=FR'" - os: ubuntu-latest - jdk: 20 + jdk: 21 + args: "" runs-on: ${{ matrix.os }} timeout-minutes: 15 steps: @@ -71,10 +72,10 @@ jobs: - uses: actions/checkout@v4 with: submodules: true - - name: Set up JDK 17 + - name: Set up JDK 21 uses: actions/setup-java@v3 with: - java-version: 17 + java-version: 21 distribution: 'temurin' - name: Build and test run: mvn --batch-mode -no-transfer-progress package --file standalone.pom.xml @@ -102,7 +103,7 @@ jobs: - name: Set up JDK uses: actions/setup-java@v3 with: - java-version: 17 + java-version: 21 distribution: 'temurin' cache: 'maven' @@ -139,7 +140,7 @@ jobs: - name: Set up JDK uses: actions/setup-java@v3 with: - java-version: 17 + java-version: 21 distribution: 'temurin' cache: 'maven' diff --git a/.github/workflows/performance.yml b/.github/workflows/performance.yml index 042cba5303..bba3960063 100644 --- a/.github/workflows/performance.yml +++ b/.github/workflows/performance.yml @@ -44,7 +44,7 @@ jobs: - name: Set up JDK uses: actions/setup-java@v3 with: - java-version: 17 + java-version: 21 distribution: 'temurin' cache: 'maven' - uses: actions/setup-node@v4 diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index e94bb71210..5d1968a6ed 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -18,65 +18,65 @@ jobs: contents: write packages: write steps: - - name: Ensure version does not start with 'v' - uses: actions/github-script@v6 - with: - github-token: ${{ github.token }} - script: | - version = context.payload.inputs.version; - if (/^v/.test(version)) throw new Error("Bad version number: " + version) - - uses: actions/checkout@v4 - with: - submodules: true - - name: Cache data/sources - uses: ./.github/cache-sources-action - - uses: actions/setup-java@v3 - with: - java-version: '17' - distribution: 'temurin' - cache: 'maven' - server-id: ossrh - server-username: MAVEN_USERNAME - server-password: MAVEN_PASSWORD + - name: Ensure version does not start with 'v' + uses: actions/github-script@v6 + with: + github-token: ${{ github.token }} + script: | + version = context.payload.inputs.version; + if (/^v/.test(version)) throw new Error("Bad version number: " + version) + - uses: actions/checkout@v4 + with: + submodules: true + - name: Cache data/sources + uses: ./.github/cache-sources-action + - uses: actions/setup-java@v3 + with: + java-version: '21' + distribution: 'temurin' + cache: 'maven' + server-id: ossrh + server-username: MAVEN_USERNAME + server-password: MAVEN_PASSWORD - - name: Check tag does not exist yet - run: if git rev-list "v${{ github.event.inputs.version }}"; then echo "Tag already exists. Aborting the release process."; exit 1; fi + - name: Check tag does not exist yet + run: if git rev-list "v${{ github.event.inputs.version }}"; then echo "Tag already exists. Aborting the release process."; exit 1; fi - - run: ./scripts/set-versions.sh "${{ github.event.inputs.version }}" - - run: ./scripts/build-release.sh - - run: ./scripts/test-release.sh "${{ github.event.inputs.version }}" - - name: Create tag - uses: actions/github-script@v6 - with: - github-token: ${{ github.token }} - script: | - github.rest.git.createRef({ - owner: context.repo.owner, - repo: context.repo.repo, - ref: "refs/tags/v${{ github.event.inputs.version }}", - sha: context.sha - }) - - run: mv planetiler-dist/target/*with-deps.jar planetiler.jar - - run: sha256sum planetiler.jar > planetiler.jar.sha256 - - run: md5sum planetiler.jar > planetiler.jar.md5 - - name: Install GPG Private Key - run: | - echo -n "${{ secrets.OSSRH_GPG_SECRET_KEY }}" | base64 --decode | gpg --batch --import - - name: Create Release - uses: softprops/action-gh-release@v1 - with: - fail_on_unmatched_files: true - tag_name: v${{ github.event.inputs.version }} - draft: true - files: | - planetiler.jar* - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - - run: ./scripts/push-release.sh ${{ github.event.inputs.version }} - env: - GITHUB_ACTOR: ${{ github.actor }} - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - IMAGE_TAGS: ${{ github.event.inputs.image_tags }} - MAVEN_USERNAME: ${{ secrets.OSSRH_USERNAME }} - MAVEN_PASSWORD: ${{ secrets.OSSRH_PASSWORD }} - OSSRH_GPG_SECRET_KEY_PASSWORD: ${{ secrets.OSSRH_GPG_SECRET_KEY_PASSWORD }} + - run: ./scripts/set-versions.sh "${{ github.event.inputs.version }}" + - run: ./scripts/build-release.sh + - run: ./scripts/test-release.sh "${{ github.event.inputs.version }}" + - name: Create tag + uses: actions/github-script@v6 + with: + github-token: ${{ github.token }} + script: | + github.rest.git.createRef({ + owner: context.repo.owner, + repo: context.repo.repo, + ref: "refs/tags/v${{ github.event.inputs.version }}", + sha: context.sha + }) + - run: mv planetiler-dist/target/*with-deps.jar planetiler.jar + - run: sha256sum planetiler.jar > planetiler.jar.sha256 + - run: md5sum planetiler.jar > planetiler.jar.md5 + - name: Install GPG Private Key + run: | + echo -n "${{ secrets.OSSRH_GPG_SECRET_KEY }}" | base64 --decode | gpg --batch --import + - name: Create Release + uses: softprops/action-gh-release@v1 + with: + fail_on_unmatched_files: true + tag_name: v${{ github.event.inputs.version }} + draft: true + files: | + planetiler.jar* + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + - run: ./scripts/push-release.sh ${{ github.event.inputs.version }} + env: + GITHUB_ACTOR: ${{ github.actor }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + IMAGE_TAGS: ${{ github.event.inputs.image_tags }} + MAVEN_USERNAME: ${{ secrets.OSSRH_USERNAME }} + MAVEN_PASSWORD: ${{ secrets.OSSRH_PASSWORD }} + OSSRH_GPG_SECRET_KEY_PASSWORD: ${{ secrets.OSSRH_GPG_SECRET_KEY_PASSWORD }} diff --git a/.github/workflows/snapshot.yml b/.github/workflows/snapshot.yml index 359180c6ba..88201f78b1 100644 --- a/.github/workflows/snapshot.yml +++ b/.github/workflows/snapshot.yml @@ -21,37 +21,37 @@ jobs: contents: read packages: write steps: - - uses: actions/checkout@v4 - with: - submodules: true - - name: Cache data/sources - uses: ./.github/cache-sources-action - - name: Set up JDK - uses: actions/setup-java@v3 - with: - java-version: 17 - distribution: 'temurin' - cache: 'maven' - server-id: ossrh - server-username: MAVEN_USERNAME - server-password: MAVEN_PASSWORD - - run: ./scripts/build-release.sh - - run: ./scripts/test-release.sh - - run: sha256sum planetiler-dist/target/*with-deps.jar - - run: md5sum planetiler-dist/target/*with-deps.jar - - name: 'Upload artifact' - uses: actions/upload-artifact@v3 - with: - name: planetiler-build - path: planetiler-dist/target/*with-deps.jar - - name: Install GPG Private Key - run: | - echo -n "${{ secrets.OSSRH_GPG_SECRET_KEY }}" | base64 --decode | gpg --batch --import - - run: ./scripts/push-release.sh - env: - GITHUB_ACTOR: ${{ github.actor }} - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - IMAGE_TAGS: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.image_tags || 'latest,snapshot' }} - MAVEN_USERNAME: ${{ secrets.OSSRH_USERNAME }} - MAVEN_PASSWORD: ${{ secrets.OSSRH_PASSWORD }} - OSSRH_GPG_SECRET_KEY_PASSWORD: ${{ secrets.OSSRH_GPG_SECRET_KEY_PASSWORD }} + - uses: actions/checkout@v4 + with: + submodules: true + - name: Cache data/sources + uses: ./.github/cache-sources-action + - name: Set up JDK + uses: actions/setup-java@v3 + with: + java-version: 21 + distribution: 'temurin' + cache: 'maven' + server-id: ossrh + server-username: MAVEN_USERNAME + server-password: MAVEN_PASSWORD + - run: ./scripts/build-release.sh + - run: ./scripts/test-release.sh + - run: sha256sum planetiler-dist/target/*with-deps.jar + - run: md5sum planetiler-dist/target/*with-deps.jar + - name: 'Upload artifact' + uses: actions/upload-artifact@v3 + with: + name: planetiler-build + path: planetiler-dist/target/*with-deps.jar + - name: Install GPG Private Key + run: | + echo -n "${{ secrets.OSSRH_GPG_SECRET_KEY }}" | base64 --decode | gpg --batch --import + - run: ./scripts/push-release.sh + env: + GITHUB_ACTOR: ${{ github.actor }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + IMAGE_TAGS: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.image_tags || 'latest,snapshot' }} + MAVEN_USERNAME: ${{ secrets.OSSRH_USERNAME }} + MAVEN_PASSWORD: ${{ secrets.OSSRH_PASSWORD }} + OSSRH_GPG_SECRET_KEY_PASSWORD: ${{ secrets.OSSRH_GPG_SECRET_KEY_PASSWORD }} diff --git a/.github/workflows/sonar.yml b/.github/workflows/sonar.yml index 67a05c182f..7d0367daf3 100644 --- a/.github/workflows/sonar.yml +++ b/.github/workflows/sonar.yml @@ -15,66 +15,66 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 15 steps: - - uses: actions/checkout@v4 - with: - # Disabling shallow clone is recommended for improving relevancy of reporting - fetch-depth: 0 - submodules: true - - name: Set up JDK 17 - uses: actions/setup-java@v3 - with: - java-version: 17 - distribution: 'temurin' - cache: 'maven' - - name: Cache SonarCloud packages - uses: actions/cache@v3 - with: - path: ~/.sonar/cache - key: ${{ runner.os }}-sonar - restore-keys: ${{ runner.os }}-sonar - - name: Analyze with SonarCloud - run: | - mvn -Dspotless.check.skip -Pcoverage -B verify org.sonarsource.scanner.maven:sonar-maven-plugin:sonar - env: - # Needed to get some information about the pull request, if any - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - # Read-only user, use this token to link SonarLint to SonarCloud as well - SONAR_TOKEN: c2cfe8bd7368ced07e84a620b7c2487846e220eb - - name: Wait for SonarCloud API to update... - run: "sleep 10" - - name: Upload annotations on PRs - if: ${{ github.event_name == 'pull_request' }} - uses: actions/github-script@v6 - with: - github-token: ${{ github.token }} - script: | - const pr = context.payload.pull_request.number; - const url = `https://sonarcloud.io/api/issues/search?pullRequest=${pr}&s=FILE_LINE&resolved=false&sinceLeakPeriod=true&ps=100&facets=severities%2Ctypes&componentKeys=onthegomap_planetiler&organization=onthegomap&additionalFields=_all`; - console.log("Fetching " + url); - const response = await github.request(url); - console.log("Got " + JSON.stringify(response.data)); - response.data.issues.forEach(issue => { - try { - if (issue.severity === 'INFO') return; - const textRange = issue.textRange; - const rule = encodeURIComponent(issue.rule); - const message = [ - issue.message, - '', - `rule: ${issue.rule} (https://sonarcloud.io/organizations/onthegomap/rules?open=${rule}&rule_key=${rule})`, - `issue url: https://sonarcloud.io/project/issues?pullRequest=${pr}&open=${encodeURIComponent(issue.key)}&id=onthegomap_planetiler` - ].join('\n'); - const args = { - title: `${issue.severity} ${issue.type}`, - file: issue.component.replace(/^[^:]*:/, ''), - startLine: textRange.startLine, - endLine: textRange.endLine, - startColumn: textRange.startOffset, - endColumn: textRange.endOffset - }; - core.warning(message, args); - console.log(args); - } catch (e) { - core.error(`Unable to parse sonar issue: ${JSON.stringify(issue)}`); - } - }); + - uses: actions/checkout@v4 + with: + # Disabling shallow clone is recommended for improving relevancy of reporting + fetch-depth: 0 + submodules: true + - name: Set up JDK 21 + uses: actions/setup-java@v3 + with: + java-version: 21 + distribution: 'temurin' + cache: 'maven' + - name: Cache SonarCloud packages + uses: actions/cache@v3 + with: + path: ~/.sonar/cache + key: ${{ runner.os }}-sonar + restore-keys: ${{ runner.os }}-sonar + - name: Analyze with SonarCloud + run: | + mvn -Dspotless.check.skip -Pcoverage -B verify org.sonarsource.scanner.maven:sonar-maven-plugin:sonar + env: + # Needed to get some information about the pull request, if any + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + # Read-only user, use this token to link SonarLint to SonarCloud as well + SONAR_TOKEN: c2cfe8bd7368ced07e84a620b7c2487846e220eb + - name: Wait for SonarCloud API to update... + run: "sleep 10" + - name: Upload annotations on PRs + if: ${{ github.event_name == 'pull_request' }} + uses: actions/github-script@v6 + with: + github-token: ${{ github.token }} + script: | + const pr = context.payload.pull_request.number; + const url = `https://sonarcloud.io/api/issues/search?pullRequest=${pr}&s=FILE_LINE&resolved=false&sinceLeakPeriod=true&ps=100&facets=severities%2Ctypes&componentKeys=onthegomap_planetiler&organization=onthegomap&additionalFields=_all`; + console.log("Fetching " + url); + const response = await github.request(url); + console.log("Got " + JSON.stringify(response.data)); + response.data.issues.forEach(issue => { + try { + if (issue.severity === 'INFO') return; + const textRange = issue.textRange; + const rule = encodeURIComponent(issue.rule); + const message = [ + issue.message, + '', + `rule: ${issue.rule} (https://sonarcloud.io/organizations/onthegomap/rules?open=${rule}&rule_key=${rule})`, + `issue url: https://sonarcloud.io/project/issues?pullRequest=${pr}&open=${encodeURIComponent(issue.key)}&id=onthegomap_planetiler` + ].join('\n'); + const args = { + title: `${issue.severity} ${issue.type}`, + file: issue.component.replace(/^[^:]*:/, ''), + startLine: textRange.startLine, + endLine: textRange.endLine, + startColumn: textRange.startOffset, + endColumn: textRange.endOffset + }; + core.warning(message, args); + console.log(args); + } catch (e) { + core.error(`Unable to parse sonar issue: ${JSON.stringify(issue)}`); + } + }); From cd1c3fa2ad3fbef25b119caa4269181ecfd6f699 Mon Sep 17 00:00:00 2001 From: Mike Barry Date: Wed, 25 Oct 2023 21:27:14 -0400 Subject: [PATCH 3/4] format --- README.md | 4 ++-- planetiler-custommap/README.md | 5 ++--- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index f5e82477e0..95acf2a347 100644 --- a/README.md +++ b/README.md @@ -162,7 +162,7 @@ consider [contributing](#contributing) your change back for others to use! Some example runtimes for the OpenMapTiles profile (excluding downloading resources): -| Input | Version | Machine | Time | mbtiles size | Logs | +| Input | Version | Machine | Time | mbtiles size | Logs | |-------------------------------------------------------------------------------------------------------------------------------------------|---------|---------------------------------|---------------------------------|--------------|------------------------------------------------------------------------------------------------------------------------| | s3://osm-pds/2022/planet-220530.osm.pbf (69GB) | 0.5.0 | c2d-standard-112 (112cpu/448GB) | 37m cpu:48h5m gc:3m45s avg:76.9 | 79GB | [logs](planet-logs/v0.5.0-planet-c2d-standard-112.txt) | | s3://osm-pds/2022/planet-220530.osm.pbf (69GB) | 0.5.0 | c6gd.16xlarge (64cpu/128GB) | 53m cpu:41h58m avg:47.1 | 79GB | [logs](planet-logs/v0.5.0-planet-c6gd-128gb.txt), [VisualVM Profile](planet-logs/v0.5.0-planet-c6gd-128gb.nps) | @@ -173,7 +173,7 @@ Some example runtimes for the OpenMapTiles profile (excluding downloading resour Merging nearby buildings at z13 is very expensive, when run with `--building-merge-z13=false`: -| Input | Version | Machine | Time | mbtiles size | Logs | +| Input | Version | Machine | Time | mbtiles size | Logs | |------------------------------------------------|---------|----------------------------------------------------------|--------------------------|--------------|------------------------------------------------------------------------------------------------------------------------------------------------------------| | s3://osm-pds/2022/planet-220530.osm.pbf (69GB) | 0.5.0 | c2d-standard-112 (112cpu/448GB) | 26m cpu:27h47m avg:63.9 | 79GB | [logs](planet-logs/v0.5.0-planet-c2d-standard-112-no-z13-building-merge.txt) | | s3://osm-pds/2022/planet-220530.osm.pbf (69GB) | 0.5.0 | c6gd.16xlarge (64cpu/128GB) | 39m cpu:27h4m avg:42.1 | 79GB | [logs](planet-logs/v0.5.0-planet-c6gd-128gb-no-z13-building-merge.txt), [VisualVM Profile](planet-logs/v0.5.0-planet-c6gd-128gb-no-z13-building-merge.nps) | diff --git a/planetiler-custommap/README.md b/planetiler-custommap/README.md index 946df08a13..81624a4114 100644 --- a/planetiler-custommap/README.md +++ b/planetiler-custommap/README.md @@ -144,7 +144,6 @@ to regenerate: cat planetiler-custommap/planetiler.schema.json | jq -r '.properties.args.properties | to_entries[] | "- `" + .key + "` - " + .value.description' | pbcopy --> - - `threads` - Default number of threads to use. - `write_threads` - Default number of threads to use when writing temp features - `process_threads` - Default number of threads to use when processing input features @@ -480,7 +479,7 @@ nested, so each child context can also access the variables from its parent. >> ##### process feature context >> >> Context available when processing an input feature, for example testing whether to include it from `include_when`. -> > Available variables: +>> Available variables: >> >> - `feature.tags` - map with key/value tags from the input feature >> - `feature.id` - numeric ID of the input feature @@ -497,7 +496,7 @@ nested, so each child context can also access the variables from its parent. >>>> ##### configure attribute context >>>> >>>> Context available after the value of an attribute has been computed, for example: set min zoom to render an -> > > > attribute. Adds variables: +>>>> attribute. Adds variables: >>>> >>>> - `value` the value that was computed for this key From 90d178d3273dfae63c247d37774fd712246f197c Mon Sep 17 00:00:00 2001 From: Mike Barry Date: Wed, 25 Oct 2023 22:01:00 -0400 Subject: [PATCH 4/4] better error message when using old version --- planetiler-dist/pom.xml | 6 ++++++ .../src/main/java/com/onthegomap/planetiler/Main.java | 9 +++++++++ 2 files changed, 15 insertions(+) diff --git a/planetiler-dist/pom.xml b/planetiler-dist/pom.xml index af6ef5335b..7af1b23c7a 100644 --- a/planetiler-dist/pom.xml +++ b/planetiler-dist/pom.xml @@ -14,6 +14,12 @@ com.onthegomap.planetiler.Main + + 17 + 17 ${project.version} ghcr.io/onthegomap/planetiler:${image.version} package diff --git a/planetiler-dist/src/main/java/com/onthegomap/planetiler/Main.java b/planetiler-dist/src/main/java/com/onthegomap/planetiler/Main.java index 3a8fcf2732..df949ed305 100644 --- a/planetiler-dist/src/main/java/com/onthegomap/planetiler/Main.java +++ b/planetiler-dist/src/main/java/com/onthegomap/planetiler/Main.java @@ -25,6 +25,15 @@ * public static void main(String[] args)} methods of runnable classes. */ public class Main { + static { + int version = Runtime.version().feature(); + if (version < 21) { + System.err.println( + "You are using Java " + version + + " but Planetiler requires 21 or later, for more details on upgrading see: https://github.com/onthegomap/planetiler/blob/main/CONTRIBUTING.md"); + System.exit(1); + } + } private static final EntryPoint DEFAULT_TASK = OpenMapTilesMain::main; private static final Map ENTRY_POINTS = Map.ofEntries(