modlib/Readme.md

424 lines
12 KiB
Markdown
Raw Normal View History

2021-09-10 21:06:16 +02:00
# ![Logo](logo.svg) Modding Library (`modlib`)
2020-02-13 11:11:52 +01:00
Multipurpose Minetest Modding Library
2019-05-01 16:53:01 +02:00
2019-05-03 13:51:17 +02:00
## About
2020-02-13 11:11:52 +01:00
2021-09-24 18:30:30 +02:00
No dependencies. Licensed under the MIT License. Written by Lars Mueller aka LMD or appguru(eu). Notable contributions by [luk3yx](https://github.com/luk3yx) in the form of suggestions, bug reports and fixes.
2019-05-03 13:51:17 +02:00
## Tests
The tests are located in a different repo, [`modlib_test`](https://github.com/appgurueu/modlib_test), as they are quite heavy due to testing the PNG reader using PngSuite. Reading the tests for examples of API usage is recommended.
2019-05-01 16:53:01 +02:00
## API
2020-02-13 11:11:52 +01:00
2021-08-16 18:11:55 +02:00
Mostly self-documenting code. Mod namespace is `modlib`, containing all variables & functions.
2021-04-21 21:11:00 +02:00
### Persistence
#### Lua Log Files
A data log file based on Lua statements. **Experimental.** High performance. Example from `test.lua`:
```lua
local logfile = persistence.lua_log_file.new(mod.get_resource"logfile.test.lua", {})
logfile:init()
logfile.root = {}
logfile:rewrite()
logfile:set_root({a = 1}, {b = 2, c = 3})
logfile:close()
logfile:init()
assert(table.equals(logfile.root, {[{a = 1}] = {b = 2, c = 3}}))
```
2021-07-03 19:35:02 +02:00
Both strings and tables are stored in a reference table. Unused strings won't be garbage collected as Lua doesn't allow marking them as weak references.
This means that setting lots of temporary strings will waste memory until you call `:rewrite()` on the log file. An alternative is to set the third parameter, `reference_strings`, to `false` (default value is `true`):
```lua
persistence.lua_log_file.new(mod.get_resource"logfile.test.lua", {}, false)
```
This will prevent strings from being referenced, possibly bloating file size, but saving memory.
2021-07-14 21:52:13 +02:00
#### SQLite3 Database Persistence
2022-02-27 17:43:05 +01:00
Uses a SQLite3 database to persistently store a Lua table. **Experimental.** Obtaining it is a bit trickier, as it requires access to the `lsqlite3` library, which may be passed:
2021-07-14 21:52:13 +02:00
```lua
local modlib_sqlite3 = persistence.sqlite3(require"lsqlite3")
```
(assuming `require` is that of an insecure environment if Minetest is used)
Alternatively, if you are not running Minetest, mod security is disabled, you have (temporarily) provided `require` globally, or added `modlib` to `secure.trusted_mods`, you can simply do the following:
```lua
local modlib_sqlite3 = persistence.sqlite3()
```
Modlib will then simply call `require"lsqlite3"` for you.
Then, you can proceed to create a new database:
```lua
local database = persistence.modlib_sqlite3.new(mod.get_resource"database.test.sqlite3", {})
-- Create or load
database:init()
-- Use it
database:set_root("key", {nested = true})
database:close()
```
It uses a similar API to Lua log files:
* `new(filename, root)` - without `reference_strings` however (strings aren't referenced currently)
* `init`
* `set`
* `set_root`
* `rewrite`
* `close`
The advantage over Lua log files is that the SQlite3 database keeps disk usage minimal. Unused tables are dropped from the database immediately through reference counting. The downside of this is that this, combined with the overhead of using SQLite3, of course takes time, making updates on the SQLite3 database slower than Lua log file updates (which just append to an append-only file).
As simple and fast reference counting doesn't handle cycles, an additional `collectgarbage` stop-the-world method performing a full garbage collection on the database is provided which is called during `init`.
The method `defragment_ids` should not have to be used in practice (if it has to be, it happens automatically) and should be used solely for debugging purposes (neater IDs).
2021-03-05 15:08:57 +01:00
### Bluon
Binary Lua object notation. **Experimental.** Handling of subnormal numbers (very small floats) may be broken.
#### `new(def)`
```lua
def = {
2021-07-14 11:51:47 +02:00
aux_is_valid = function(object)
return is_valid
end,
aux_len = function(object)
return length_in_bytes
end,
-- read type byte, stream providing :read(count), map of references -> id
aux_read = function(type, stream, references)
... = stream:read(...)
return object
end,
-- object to be written, stream providing :write(text), list of references
aux_write = function(object, stream, references)
stream:write(...)
end
2021-03-05 15:08:57 +01:00
}
```
#### `:is_valid(object)`
Returns whether the given object can be represented by the instance as boolean.
#### `:len(object)`
Returns the expected length of the object if serialized by the current instance in bytes.
#### `:write(object, stream)`
Writes the object to a stream supporting `:write(text)`. Throws an error if invalid.
#### `:read(stream)`
Reads a single bluon object from a stream supporting `:read(count)`. Throws an error if invalid bluon.
Checking whether the stream has been fully consumed by doing `assert(not stream:read(1))` is left up to the user.
#### Format
* `nil`: nothing (`""`)
* `false`: 0
* `true`: 1
* Numbers:
* Constants: 0, nan, +inf, -inf
* Integers: Little endian `U8`, `U16`, `U32`, `U64`, `-U8`, `-U16`, `-U32`, `-U64`
* Floats: Little endian `F32`, `F64`
* Strings:
* Constant: `""`
* Length as unsigned integer: `T8`, `T16`, `T32`, `T64`
* Tables:
* List and map part count as unsigned integers
* `L0`, `L8`, `L16`, `L32`, `L64` times `M0`, `M8`, `M16`, `M32`, `M64`
* Reference:
* Reference ID as unsigned integer: `R8`, `R16`, `R32`, `R64`
* Reserved types:
* Everything <= 55 => 200 free types
#### Features
* Embeddable: Written in pure Lua
* Storage efficient: No duplication of strings or reference-equal tables
* Flexible: Can serialize circular references and strings containing null
#### Simple example
```lua
local object = ...
-- Write to file
local file = io.open(..., "wb")
2021-03-05 15:08:57 +01:00
modlib.bluon:write(object, file)
file:close()
-- Write to text
local rope = modlib.table.rope{}
modlib.bluon:write(object, rope)
text = rope:to_text()
-- Read from text
local inputstream = modlib.text.inputstream"\1"
assert(modlib.bluon:read(object, rope) == true)
```
#### Advanced example
```lua
-- Serializes all userdata to a constant string:
local custom_bluon = bluon.new{
2021-07-14 11:51:47 +02:00
aux_is_valid = function(object)
return type(object) == "userdata"
end,
aux_len = function(object)
return 1 + ("userdata"):len())
end,
aux_read = function(type, stream, references)
assert(type == 100, "unsupported type")
assert(stream:read(("userdata"):len()) == "userdata")
return userdata()
end,
-- object to be written, stream providing :write(text), list of references
aux_write = function(object, stream, references)
assert(type(object) == "userdata")
stream:write"\100userdata"
end
2021-03-05 15:08:57 +01:00
}
-- Write to text
local rope = modlib.table.rope{}
custom_bluon:write(userdata(), rope)
assert(rope:to_text() == "\100userdata")
```
### Schema
Place a file `schema.lua` in your mod, returning a schema table.
#### Non-string entries and `minetest.conf`
Suppose you have the following schema:
```lua
return {
type = "table",
entries = {
[42] = {
type = "boolean",
description = "The Answer"
default = true
}
}
}
```
And a user sets the following config:
```conf
mod.42 = false
```
It won't work, as the resulting table will be `{["42"] = false}` instead of `{[42] = false}`. In order to make this work, you have to convert the keys yourself:
```lua
return {
type = "table",
keys = {
-- this will convert all keys to numbers
type = "number"
},
entries = {
[42] = {
type = "boolean",
description = "The Answer"
default = true
}
}
}
```
This is best left explicit. First, you shouldn't be using numbered field keys if you want decent `minetest.conf` support, and second, `modlib`'s schema module could only guess in this case, attempting conversion to number / boolean. What if both number and string field were set as possible entries? Should the string field be deleted? And so on.
## Configuration
2020-12-20 14:39:01 +01:00
### Legacy
1. Configuration is loaded from `<worldpath>/config/<modname>.<extension>`, the following extensions are supported and loaded (in the given order), with loaded configurations overriding properties of previous ones:
1. [`json`](https://json.org)
2. [`lua`](https://lua.org)
3. [`luon`](https://github.com/appgurueu/luon), Lua but without the `return`
4. [`conf`](https://github.com/minetest/minetest/blob/master/doc/lua_api.txt)
2020-12-20 14:39:01 +01:00
2. Settings are loaded from `minetest.conf` and override configuration values
### Locations
0. Default configuration: `<modfolder>/conf.lua`
1. World configuration: `config/<modname>.<format>`
2. Mod configuration: `<modfolder>/conf.<format>`
3. Minetest configuration: `minetest.conf`
### Formats
1. [`lua`](https://lua.org)
* Lua, with the environment being the configuration object
* `field = value` works
* Return new configuration object to replace
2. [`luon`](https://github.com/appgurueu/luon)
* Single Lua literal
* Booleans, numbers, strings and tables
3. [`conf`](https://github.com/minetest/minetest/blob/master/doc/lua_api.txt)
* Minetest-like configuration files
4. [`json`](https://json.org)
2021-03-30 18:10:50 +02:00
* Not recommended
2021-05-04 16:18:46 +02:00
### `variables(stacklevel)`
Dumps local variables, upvalues and the function environment of the function at the given stacklevel (default `1`).
### `stack(stacklevel)`
Dumps function info & variables for all functions in stack, starting with stacklevel (default `1`).
2021-05-08 21:17:09 +02:00
## `minetest`
### `schematic`
A schematic format with support for metadata and baked light data. **Experimental.**
2021-04-23 19:35:03 +02:00
## Release Notes
2022-07-03 19:00:12 +02:00
Rolling releases (corresponding to releases on ContentDB) are made when either:
* A fix for non-experimental, in-use functionality is implemented
* A sufficient amount of features is published
The following release notes are *not exhaustive*. Refer to the commit log for an exhaustive changelog.
### `rolling-98`
* Fixes: `binary.(read|write)_(float|single|double)` (and thus also `bluon` number serialization)
* `write_float` now doesn't provide an "auto" mode anymore, use `modlib.math.fround(x) == x` instead to determine whether `x` fits in a float
* Removal of the `debug` module in favor of the [`dbg`](https://github.com/appgurueu/dbg) mod
### `rolling-97`
* Removal of tests in favor of a dedicated test repo
2022-06-08 11:21:33 +02:00
### `rolling-96`
* Additions: B3D writer; `mod.configuration()` returns the `conf, schema` now
* Fixes: `binary`: int & float writing
### `rolling-9x`
Many many fixes
### `rolling-8x`
Fixes, minor additions (such as a `hashheap`)
2021-09-15 19:14:34 +02:00
### `rolling-74`
* Fixes: Plenty, mostly related to the `minetest` module. Note:
* `table.shuffle` was fixed (previously had an off-by-one error); small tables are now shuffled correctly
* A crash on loading in the `minetest` module was fixed (note that this crash was not included in the `rolling-73` release)
* Additions:
* A logo
* `minetest` module:
* Priority queue based `after`
* `playerdata`, `connected_players` & `set_privs` utilities
* Experimental `get_mod_info` & `get_mod_load_order`, `media` path getting
* Lazy loading of components
* `quaternion.from_euler_rotation`
* `text.trim_spacing`
* Experimental `json` module
* Removal of `modlib.minetest.get_color_int`
2021-08-15 14:33:38 +02:00
### `rolling-73`
* Fixes Mesecons LuaController overheating by luk3yx
2021-07-12 16:29:36 +02:00
### `rolling-71`
* Fixes
* Colorspec
2021-07-14 11:51:47 +02:00
* Stricter patterns in `colorspec:from_string`
* `colorspec:to_string` works now
2021-07-12 16:29:36 +02:00
* Lua log file
2021-07-14 11:51:47 +02:00
* Patched memory leaks
* Added option to not reference strings
* Handling of circular tables
2021-07-12 16:29:36 +02:00
* Additions
* `luon`: Configurable serialization to and from Lua with circular and string reference support
2021-07-14 11:51:47 +02:00
* `minetest.luon` even supports `ItemStack` and `AreaStore`
2021-07-12 16:29:36 +02:00
* `vector.rotate3`
* `table.deep_foreach_any`
* `func`:
2021-07-14 11:51:47 +02:00
* `func.iterate`
* `func.aggregate`
* Functional wrappers for operators
2021-07-12 16:29:36 +02:00
* Improvements
* Code quality
* Performance
* Proper license file
2021-06-18 20:58:21 +02:00
### `rolling-70`
* Fixes module environments once and for all
* Fixes vector aliases
* Presumably boosts performance
2021-05-13 15:15:04 +02:00
### `rolling-69`
* Fixes various things, **most importantly modules indexing the global table**
2021-04-23 19:35:03 +02:00
### `rolling-68`
* Replace changelog by release notes (see the commit log for changes)
2021-03-30 18:10:50 +02:00
2021-04-23 12:43:05 +02:00
### `rolling-67`
* Fixes various things, **most importantly objects indexing the global table**
* Concerns `kdtree`, `trie`, `ranked_set`, `vector`, `schema`
2021-04-21 21:40:07 +02:00
### `rolling-66`
* Adds `modlib.persistence.lua_log_file`
2021-04-02 21:09:40 +02:00
### `rolling-62`
* Fix `modlib.func.curry_tail`
* Change `b3d:get_animated_bone_properties` to return a list according to hierarchy
2021-04-01 01:06:12 +02:00
### `rolling-61`
* Fix `quaternion.to_euler_rotation`
2021-04-01 00:05:34 +02:00
### `rolling-60`
* Fix `vector.interpolate`
2021-03-31 23:42:41 +02:00
### `rolling-59`
* Schema failing check handling fixes
2021-03-31 22:37:04 +02:00
### `rolling-58`
* Schema improvements & docs
2021-03-30 22:27:16 +02:00
### `rolling-57`
* Uses `minetest.safe_file_write` for `file.write`
2021-03-30 18:10:50 +02:00
### `rolling-56`
* Fixes `math.fround`
2021-03-30 18:12:25 +02:00
* Other minor fixes
* Switch to lazy loading
* Do `_ = modlib.<module>` to avoid lag spikes at run time