directory structure for pumpkin files

[KairuDeibisu]

I did not mean the data volume. (The volume should be renamed to pumpkin) All things that changed should be a separate folder like data and not inside the root folder.

The goal is to create a robust folder structure. Maybe I should make this an issue so it can be discussed on properly.

• pumpkin(.exe) (can be read and execute only)
• config (you could now make this folder red only)
  • configuration.toml
  • features.toml
• plugins (you could now make this folder read and execute only)
  • example(.dll)
  • example2(.dll)
  • example3(.dll)
• data (needs to be rw to work)
  • world
  • ops.json
  • (maby) perms.db

A folder structure like this is awesome if you use docker because now you can create volumes with restricting permissions.

Originally posted by @Commandcracker in #348 (comment)

[KairuDeibisu]

I figured it'd be better to open this up as a wider discussion. Anybody have any ideas on what we could do for the directory structure?

[vyPal]

Usually plugins need to store data somewhere, for this I'd copy Bukkit, that each plugin has it's own separate folder inside the plugins folder, but this would mean that the plugins folder could no longer just be r-x but would need wow as well

[Commandcracker]

Something like this ?

• 📄 pumpkin [RX]
• 📁 config [R]
  • 📄 config.toml
  • 📄 advanced.toml
  • 📁 plugins
    • 📁 example
      • 📄 config.toml
    • 📁 example2
      • 📄 config.toml
    • 📁 example3
      • 📄 config.toml
• 📁 plugins [RX]
  • 📄 example
  • 📄 example2
  • 📄 example3
• 📁 data [RW]
  • 📁 plugins
    • 📁 example
      • 📄 some_data.json
    • 📁 example2
      • 📄 some_data.json
    • 📁 example3
      • 📄 some_data.json
  • 📁 worlds (maby move to root ?)
    • 📁 world
    • 📁 world_nether (do we want to have nether seperat)
    • 📁 world_the_end (do we want to have end seperat)
    • 📁 world2
  • 📄 ops.json
  • 📄 whitelist.json
  • 📄 banned-ips.json
  • 📄 banned-players.json
  • 📁 logs (maby move to root ?)

[Snowiiii]

Something like this ?

• 📄 pumpkin [RX]

• 📁 config [R]

  • 📄 config.toml

  • 📄 advanced.toml

  • 📁 plugins

    • 📁 example

      • 📄 config.toml

    • 📁 example2

      • 📄 config.toml

    • 📁 example3

      • 📄 config.toml

• 📁 plugins [RX]

  • 📄 example
  • 📄 example2
  • 📄 example3

• 📁 data [RW]

  • 📁 plugins

    • 📁 example

      • 📄 some_data.json

    • 📁 example2

      • 📄 some_data.json

    • 📁 example3

      • 📄 some_data.json

  • 📁 worlds (maby move to root ?)

    • 📁 world
    • 📁 world_nether (do we want to have nether seperat)
    • 📁 world_the_end (do we want to have end seperat)
    • 📁 world2

  • 📄 ops.json

  • 📄 whitelist.json

  • 📄 banned-ips.json

  • 📄 banned-players.json

  • 📁 logs (maby move to root ?)

I like the structure. We only thing i see is that we may want to move the logs
folder out of data/ because it needs to be write only

[DaniD3v]

Is there a reason we're putting everything in 1 folder?
Why not just follow linux spec l and make a /etc/pumpkin /var/pumpkin etc?

ig this would lead to cross compatibility issues. There are libraries for handling fhs/xdg dirs crossplatforn.

[KairuDeibisu]

Is there a reason we're putting everything in 1 folder? Why not just follow linux spec l and make a /etc/pumpkin /var/pumpkin etc?

ig this would lead to cross compatibility issues. There are libraries for handling fhs/xdg dirs crossplatforn.

A self-contained directory simplifies deployment and backup, especially for users on non-Linux systems or with minimal technical knowledge. Minecraft is a game for all ages. We need to keep in mind our demographic.

[DataM0del]

Is there a reason we're putting everything in 1 folder? Why not just follow linux spec l and make a /etc/pumpkin /var/pumpkin etc?
ig this would lead to cross compatibility issues. There are libraries for handling fhs/xdg dirs crossplatforn.

A self-contained directory simplifies deployment and backup, especially for users on non-Linux systems or with minimal technical knowledge. Minecraft is a game for all ages. We need to keep in mind our demographic.

Well then just add a symlink to it in the folder where the build is.

[Commandcracker]

Is there a reason we're putting everything in 1 folder? Why not just follow linux spec l and make a /etc/pumpkin /var/pumpkin etc?

ig this would lead to cross compatibility issues. There are libraries for handling fhs/xdg dirs crossplatforn.

This would be very messy with multiple minecraft servers on one machine.

[NoSQLKnowHow]

Question. Is there a possibility to deviate from the common Minecraft format for files like whitelist.json, banned-ips, banned-players, so that they are JSON line format instead of an array of objects. I ask this as if they are an array, I am fairly sure you have to read the entire document just to append one new entry. If instead each line was a new object, you just append to the file and keep going. This would, in theory be more efficient. There are really two times you need to read the whitelist, banned-IP, and banned-players. 1/ on init of the server and 2/ when you need to add a new entity to be listed or banned. Shifting to the JSON line format would be efficient for both of these times over the regular JSON format.

Or is this such a non-issue performance-wise that deviating from the existing format that Minecraft servers make today is not worth it?

I am asking because I am thinking of helping my son contribute to implementing the whitelist, banned-ip, and/or banned-players functionality at the console.

[Commandcracker]

Question. Is there a possibility to deviate from the common Minecraft format for files like whitelist.json, banned-ips, banned-players, so that they are JSON line format instead of an array of objects. I ask this as if they are an array, I am fairly sure you have to read the entire document just to append one new entry. If instead each line was a new object, you just append to the file and keep going. This would, in theory be more efficient. There are really two times you need to read the whitelist, banned-IP, and banned-players. 1/ on init of the server and 2/ when you need to add a new entity to be listed or banned. Shifting to the JSON line format would be efficient for both of these times over the regular JSON format.

Or is this such a non-issue performance-wise that deviating from the existing format that Minecraft servers make today is not worth it?

I am asking because I am thinking of helping my son contribute to implementing the whitelist, banned-ip, and/or banned-players functionality at the console.

Currently, we just want tings like op, whitelist, etc. to work, so we just thought of using JSON, but we will definitely move away from common minecraft server file formats to improve performance before the first release goes live. There were also discussions about using a db and I think we will make it possible to change how data is stored, similar to LuckPerms.

The JSON line format seams to be an easy way to improve performance without adding any complexity, so I would say go for it :D

A little side note: There were people who ran pumpkin on a "Raspberry pi 400", "Raspberry PI zero 2 W" and a "Luckfox Pico Max", we hope we can still support hardware like this in the future, so every performance improvement is welcome.

[NoSQLKnowHow]

I am hesitant to throw into the mix a DB like that one that looks to run on Java. SQLite, IMO, would likely be a better choice as it's ubiquitous, has indexes, written in C++, etc.

That said anything we can do to avoid loading up files into memory when we don't need the whole file, is best. I need to look deeper if SQLite loads its data files into memory. The big one to avoid is having to avoid loading into memory when all you want to do is add a new entry. JSON lines format does that, a database may not. The other time the file/DB would be used is on read when the server starts or each time when someone logs into the game.

Regardless, let me do some testing and present results.

[NoSQLKnowHow]

The other reason I am partial to SQLite over JSON files is since this will be multi-threaded and support thousands of users, concurrency and atomicity become an issue with JSON files. In SQLite, this is a non-issue. In addition it only loads into memory the page it needs for the query. So if I design a schema optimized for logging in users, ie are they banned, are they white listed, is their IP whitelisted, and make sure to use covered indexes, SQLite will be SUPER efficient and very lite on memory footprint.

Regardless, let me work up an initial ERD so we have something to look at and discuss.

The other thing is what else should be persisted into a SQLite database beside whitelists and ban lists? Again, if we're looking for high concurrency and efficient, low memory usage, the more I think about it, the more json files are not a solution.

[Commandcracker]

I also had a db like SQLite in the back of my mind, but if a database is implemented, it should support the flexibility to switch between different database drivers.

Things we should consider while designing are:

• Server Types
  • big single servers
  • server networks (Velocity, Bungee)
  • possible support for server sharding / server meshing, see https://multipaper.io/howitworks.html
• Minecraft account types
  • online
  • cracked
  • possible support for bedrock accounts, see https://geysermc.org/
• Tools for Importing and Exporting to vanilla (also: bucket, spigot, paper) minecraft server
• Migration (Upgrade and Downgrade) see https://alembic.sqlalchemy.org/en/latest/
• Different and new minecraft versions when storing items, blocks or entities

The other thing is what else should be persisted into a SQLite database beside whitelists and ban lists?

• [Discussion] World persistence over Database #92 worlds (including player data like advancements, position, health, stats, and much more)
• Permission system #35 permissions
• ops
• banned-players
• banned-ips
• whitelist
• maybe also allow plugins to store data in it

[NoSQLKnowHow]

@Commandcracker for your list under "Server Types" to kinds of considerations start getting into more serious database. Which is fine, but my concern is that people who want to run a Minecraft server that scales all of a sudden now need to admin a database. I am thinking there is a balance to be struck here between the performance and scalability needs, and simplicity needs.

That said the one link talks about using LMDB, which is a KV NoSQL DB, and I am quite fond of NoSQL DBs, but what I guess I am wondering is, what are the targets for scaling this server to support player wise? IMO, the use case and targets should drive which database is used and why. Knowing this dictates the answers to the bullets you have under "Server Types" and other places.

[Commandcracker]

Hypixel the biggest minecraft server is using MongoDB and Redis. (see hypixel dev-blogs)

Well, I think we should just use SQLite as default. There is no extra administration needed for small servers and because it is SQL big servers can just hook up any db and almost all big servers already have some SQL db running.

We should just make it possible to select which storage method should be used, file based or other databases, then the server owner can choose.

[Snowiiii]

The configuration files are now stored in a separate folder, See #407