Beginnings of documentation, & some minor occasional cleanup

Basically, need to finish the User Libraries and User Space sections.

The entirety of kernel space should be documented now at least.
This commit is contained in:
20kdc 2018-04-04 02:48:10 +01:00
parent d16a9602f3
commit e984f97ea9
13 changed files with 798 additions and 15 deletions

View File

@ -15,16 +15,13 @@
-- though with automatically closing windows on process death.
-- How Bristol talks to this is:
-- 1. Bristol starts up Everest. Everest does not claim new monitors by default.
-- 2. Bristol claims all available monitors to blank out the display
-- 3. The user logs in
-- 4. Bristol runs "startSession", enabling claiming of free monitors, and then promptly dies.
-- 1. The user logs in
-- 2. Bristol starts up Everest, and frees the primary monitor
-- 3. The primary monitor is claimed by Everest and becomes monitor 1
-- 4. After a small time, Bristol dies, unclaiming all monitors
-- 5. Everest claims the new monitors, and the desktop session begins
-- 6. Everest dies/respawns, or endSession is called - in both cases,
-- Everest is now essentially back at the state in 1.
-- 7. Either this is Bristol, so go to 2,
-- or this is a screensaver host, and has a saving-throw to start Bristol if it dies unexpectedly.
-- In any case, this eventually returns to 2 or 4.
-- 6. Everest shuts down for some reason,
-- sys-init gets started UNLESS endSession(false) was used
local everestProvider = neo.requireAccess("r.neo.pub.window", "registering npw")
local everestSessionProvider = neo.requireAccess("r.neo.sys.session", "registering nsse")
@ -46,7 +43,7 @@ neo.requestAccess("s.h.key_down")
local monitors = {}
-- NULL VIRTUAL MONITOR!
-- This is where we stuff processes while Bristol isn't online
-- This is where we stuff processes until monitors show up
monitors[0] = {nil, nil, 160, 50}
-- {monitor, x, y, w, h, callback}

View File

@ -14,7 +14,6 @@ local gpus = neo.requireAccess("c.gpu", "screen control")
local screens = neo.requireAccess("c.screen", "screen control")
neo.requireAccess("s.h.component_added", "HW management")
neo.requireAccess("s.h.component_removed", "HW management")
neo.requireAccess("s.h.key_down", "Keymap guesswork")
local function shutdownFin(reboot)
-- any final actions donkonit needs to take here

View File

@ -1,7 +1,7 @@
return {
["neo"] = {
desc = "KittenOS NEO Kernel & Base Libs",
v = 1,
v = 2,
deps = {
},
dirs = {
@ -50,7 +50,7 @@ return {
},
["neo-everest"] = {
desc = "KittenOS NEO / Everest (windowing)",
v = 0,
v = 2,
deps = {
"neo"
},

View File

@ -263,7 +263,7 @@ wrapOs = wrapMeta({
totalMemory = computer.totalMemory, freeMemory = computer.freeMemory,
energy = computer.energy, maxEnergy = computer.maxEnergy,
clock = os.clock, date = os.date, difftime = os.difftime,
time = os.time, uptime = computer.uptime
time = os.time, uptime = computer.uptime, address = computer.address
})
wrapDebug = wrapMeta(debug)
@ -530,7 +530,7 @@ function start(pkg, ...)
env.neo.requestAccessAsync = requestAccessAsync
env.neo.requestAccess = function (perm, handler)
requestAccessAsync(perm)
if not handler then handler = function() end end
handler = handler or function() end
while true do
local n = {coroutine.yield()}
handler(table.unpack(n))

View File

@ -17,5 +17,24 @@ return {
files = {
"apps/app-eeprog.lua"
},
},
["neo-docs"] = {
desc = "KittenOS NEO system documentation",
v = 2,
deps = {
},
dirs = {
"docs"
},
files = {
"docs/an-intro",
"docs/kn-intro",
"docs/kn-refer",
"docs/kn-sched",
"docs/kn-perms",
"docs/us-perms",
"docs/ul-neoux",
"docs/ul-broil",
},
}
}

245
repository/docs/an-intro Normal file
View File

@ -0,0 +1,245 @@
Welcome to the documentation for your
KittenOS NEO system.
These documents are written to a 37-
column standard, in order to match
the Neolithic text editor.
If editing them, please set a right
margin of 37, or an equivalent,
in order to ensure that the
documents do not require horizontal
scroll in order to read.
This documentation is aimed at those
who wish to develop things for the
KittenOS NEO system.
Due to the size of the system, it is
divided into several sections,
each section being a file.
This section will cover an overview
of the KittenOS NEO system.
It is an abstract overview, but one
that will give you a framework in
which the components make sense,
should you simply skip to them.
The KittenOS NEO system is divided
into three things - the kernel,
libraries, and the processes.
The kernel is always loaded at all
times, and is the root of all system
activity.
The libraries are essentially Lua
values that are kept in memory by a
weak-valued table.
This allows them to be reused,
and memory saved, if possible,
while also allowing unloading.
This is a critical memory management
technique in KittenOS NEO, as it
allows certain libraries to be
loaded only when in active use.
(This also means libraries can have
security side-effects, but they
always can in any system, arguably,
and it's worth it for the memory.)
The processes are applications and
services, that communicate with the
kernel via the NEO Kernel API, and
that communicate with others via
Lua values and tables shared between
the processes, that also constitute
a form of API.
These APIs are shared and retrieved
via Accesses - anything that causes
a permissions check is an Access.
Accesses are easy to replace in the
system, should you wish to heavily
customize the system in some way.
The ability to receive given events,
is also an Access, though all "k."
events are always accessible for
simplicity reasons.
"All components are replacable, as
long as you implement it correctly."
Regarding the KittenOS NEO system
that you are now possibly running,
it likely has 3 critical services.
I refer to these as the Trinity, just
because it seemed to fit.
The following list notes which APIs
they provide and require, but only
from those in the "x." space.
Anything else is an implementation
detail, subject to change - indeed,
these are only listed for the sake
of those who need to find code for
a given API.
-------------------------------------
Glacier: sys-glacier
* formerly sys-donkonit
Shell-independent parts of NEO, such
as screen management and settings,
and the "saving throw" recovery
mechanism.
Provides x.neo.sys.manage
x.neo.sys.screens
x.neo.pub.globals
Everest: sys-everest
This is the default shell.
Provides x.neo.sys.session
x.neo.pub.window
Requires x.neo.sys.screens
Prefers x.neo.sys.manage
Icecap: sys-icecap
Shell-dependent component that
gains k.root and uses Everest
to implement the security policy.
Provides x.neo.pub.base
Requires x.neo.sys.manage
Prefers x.neo.pub.window
-------------------------------------
The bootup process, meanwhile, is
rather simple.
The kernel first designates the
primaryDisk based on the boot disk,
via the "deprecated" computer API
that has no replacement and is thus
nowhere near safe to remove.
The primaryDisk is *the*
KittenOS NEO system disk - it holds
the system, in full.
Inelegance here falls to the greater
power of practicality, as a VFS is
an unnecessary component here.
After moving it's many globals into
place, it then immediately loads and
runs sys-init.
sys-init's job is to firstly display
the KittenOS NEO boot screen -
this is where it got it's original
name from, s-bristol (see: Plymouth)
It chooses the screen / GPU based on
the best combination it can find.
The screen that it displays on is
then noted as the primary screen.
During this boot screen, it firstly
starts Glacier, which is always a
necessary component for sys-init to
operate correctly.
It then starts all services that have
"run." entries set to "yes".
By default this means sys-icecap.
Note that sys- entries occur first.
This allows a security policy to be
installed, typically by sys-icecap,
that allows non-sys- parts to
perform useful operations.
Finally, screen control is passed to
Glacier, and sys-init resets the
screens to their login-screen-state.
If the settings daemon is not around
by this point, sys-init fails-safe
and allows login & safe-mode.
(If you happen to be able to cause a
sys-glacier error, during early boot
in a way that does not require any
permissions with other ways of
creating havoc, then, this may be a
flaw in the system security model.
The risk is considered worth the
ability to theoretically use the
system.)
If the setting "sys-init.nologin" is
set to "yes", then the login screen
is skipped.
If the password is not empty, then
a password prompt is used.
If Lua 5.2 is in use, the usual set
of instructions are replaced with a
warning to use Lua 5.3.
After the login screen finishes,
sys-init disclaims the primary
screen, and runs the shell, dictated
by the setting "sys-init.shell" -
if settings are not available, then
sys-everest is used as a guess.
Up to 5 seconds later, sys-init
confirms that "x.neo.sys.session" is
an existing API, and thus a shell is
currently running.
If this does not occur in time,
then sys-init provides the:
"That wasn't a shell. Try Safe Mode."
message, and causes a reboot.
If it does, then sys-init finally
exits, with the "Trinity" in place,
and all screens automatically
disclaimed by sys-init's quit.
(NOTE: If your particularly beady
service happens to get ahold of a
screen during startup, then that
screen is of course unaffected,
unless your service dies. This is of
course intentional in case you want
a service to control a screen.)
This should summarize the system.
Good luck. - 20kdc
All of the KittenOS NEO documentation
is released into the public domain.
No warranty is provided, implied,
or otherwise.

92
repository/docs/kn-intro Normal file
View File

@ -0,0 +1,92 @@
The KittenOS NEO Kernel,
aka "init.lua", or as I like to call
it, "KNOSKRNL", is what happens when
someone tries to write a microkernel
in Lua, and make it efficient.
Obviously, the result is not entirely
what would be expected from a kernel
at all, nevermind a microkernel.
In particular, it borrows an
important concept, specifically fast
yet secure IPC.
By which I of course mean that the
IPC consists of programs giving Lua
values to each other directly, and
the kernel giving the programs some
mechanisms to help secure this.
Not what you expected, I assume.
The "kn-" group of documents is about
the KittenOS NEO kernel.
This is specifically ONLY about the
kernel, and only about features the
kernel provides directly.
As the kernel provides many things to
everything under it, I believe this
is of great use.
It's now time for the notes about the
kernel side of the boot process.
Firstly, the startup of sys-init is
unlike any other - specifically, it
has a nil callerPid/callerPkg pair.
This is because no application ran a
function to create the process - it
was created by the kernel.
Secondly, here's what goes on in the
kernel when an Access is registered,
and when it's accessed:
1. The service requests access with
an AID starting with "r.".
2. The security policy presumably
accepts the registration.
3. A blank registration in the table
"accesses" is made immediately.
This registration always fails to
be retrieved, but exists.
4. A function is returned to reset
the registration.
5. The service calls the function,
thus the registration is now
completed.
6. The user-process requests access
with an AID starting with "x.",
everything after matching that
in the "r." registration.
7. The security policy presumably
accepts the use of that API.
8. The callback in the registration
is called.
It's first return value is sent
back to the user-process.
If it errors, then nil is given
instead (the error is not sent).
Thirdly, the security policy is set
by getting the kernel global table
with "k.root", and then changing the
global "securityPolicy".
Given this operation is only ever
performed once in typical use, and
having control over it is equivalent
to instant root, it seems fitting
that it is done this way.
(Making absolute power absolute is
also why the kernel loves globals.)
Finally, the kernel prevents those
processes that aren't "sys-" from
calling "sys-" processes.

76
repository/docs/kn-perms Normal file
View File

@ -0,0 +1,76 @@
This is the list of Accesses natively
supported by the KittenOS NEO
kernel. It does not include those
that are handled by services,
but does include the mechanism for
creating & using services.
For the services, see us-perms.
Here, "*" means that everything after
this point is considered an argument
to the access.
"c.*": Component. Returns:
list: Returns iterator over proxies.
These proxies may be set up to
be unalterable for obvious
security reasons.
For "filesystem", additionally:
primary: The primaryDisk proxy.
temporary: The RAM-FS proxy.
(These entries are included in the
above list - these fields serve
to identify the components.)
"s.*": Allows receiving a signal.
Totally useless for any signal
prefixed with "k." or "x." -
"k." is always let through,
and "x." can't be sent to you
in a situation where you
don't have permission,
under normal circumstances.
"k.root": The kernel's _ENV table.
"k.computer": The "computer" table,
with wrapMeta applied,
pullSignal removed,
and also pushSignal.
"k.kill": function (pid) to kill any
process on the system.
"r.*": Registers a service's API for
retrieval via the "x." mechanism.
Returns a:
function (function (pkg, pid, send))
While the registration is locked on
success, attempting to use it will
fail, as no handler has been given.
The returned function finishes the
registration with a callback used
for when a process tries to use the
registered API.
What that API returns goes to the
target process.
The given "sendSig" function can be
used to send an event to the target
process, the type of which matches
the "x." name of the access, and
the parameters being those given to
sendSig.
So sendSig(1) from a service
registered via "r.carrot" would
generate the event: "x.carrot", 1.
(NOTE: Regarding management of
processes that die, just make sure
to check for k.procdie events and
handle as necessary.)
"x.*": Accesses a registered service
API, returning whatever that service
intends to give you. Also gives you
"s.x.*" automatically to receive the
output of sendSig.

282
repository/docs/kn-refer Normal file
View File

@ -0,0 +1,282 @@
This is a full reference on those
functions and fields exposed by the
kernel to processes and libraries.
Firstly, it is important to note that
a process runs within a coroutine.
This allows a highly "traditional"
form of mixing async and synchronous
code with event-loop nesting and
such designs. If this is not to your
taste then you can just use one, not
nested event loop.
As it runs in a coroutine, events are
received via coroutine.yield() -
sandboxers beware! You may have to
use coroutine.running() in order to
successfully hide the implementation
details of your sandbox (also events
and potentially accesses headed in
its direction...)
An example KittenOS NEO program,
solely using kernel APIs,
that you will likely have to kill:
neo.scheduleTimer(os.uptime() + 1)
while true do
local ev = coroutine.yield()
if ev == "k.timer" then
neo.emergency("Hello...")
neo.scheduleTimer(os.uptime() + 1)
end
end
This will say "Hello..." via the
neo.emergency mechanism once every
second, independently of anything
else on the system.
While this is obviously not a sane
sys-init for actual use, if you have
a disk that you can copy the kernel
to and a copy of this, it might make
a fun experiment.
The way to exit the program is to
return from your process's main
function.
The first field to note is:
_VERSION: _VERSION from the host.
The following are just wrapMeta'd
host libraries (*: altered):
math, table, string, unicode*,
coroutine, os*, debug
unicode is extended with:
safeTextFormat(s, p):
Takes a string s, and a position p,
(the position is optional, and is
assumed to be 1 otherwise)
and returns a space-padded string,
with a space after each wide char
to make unicode.len & co. act in
screen units, along with the
position translated.
undoSafeTextFormat(s):
Takes a string in padded-widechar
format, and gets rid of the pad.
Note that if padding is *missing*,
wide characters become spaces.
This leaves a string that's usually
safe to pass to a GPU without any
odd graphical glitches.
os is replaced with:
totalMemory = computer.totalMemory,
freeMemory = computer.freeMemory,
energy = computer.energy,
maxEnergy = computer.maxEnergy,
clock = os.clock, date = os.date,
difftime = os.difftime,
time = os.time,
uptime = computer.uptime,
address = computer.address
The following are just wrapMeta'd
host functions
(*: wrapped for security):
assert, ipairs, load, next*,
pairs, pcall, xpcall, select,
type, error, tonumber, tostring,
setmetatable, getmetatable*,
rawset*, rawget, rawlen, rawequal
"require" and "neo" are the parts of
the environment where a NEO-specific
nature presents itself.
require takes a string, and returns
the value returned by the library at
"libs/" .. str .. ".lua" on the
primary disk.
The library name must be a valid path
component, and the library path must
also be valid - see
ensurePathComponent, ensurePath for
more info.
The "neo" table is where most of the
NEO-specificness is hiding, which is
probably shown by its name.
It is also where libraries differ to
processes, as libraries get a subset
of the table.
For libraries, it contains:
emergency: Equals ocemu.log, if
available on the system. Else, NOP.
readBufSize: The readBufSize kernel
configuration value. Default: 2048.
Adjusting this in the kernel allows
adjusting how much the system will
read at any given time, which can
have non-obvious memory usage
effects.
Do note, following this limit is
not a requirement and is not
enforced - it's not a security
matter, just optimization/memory.
wrapMeta: A function that takes a
value, and wraps it in such a way
as to be immutable, returning the
wrapped value.
This is the first line of defense
against memory use - by using this
to protect a table, the result can
be shared between untrusted code.
listProcs: A function that returns a
table of processes. Index is ipairs
-friendly, values are:
{pid, pkg, cpuUsageInSeconds}
listApps: Returns an ipairs-friendly
list of applications on the system,
such as:
{"app-out-of-sight-is-out-of-mind",
"svc-i-see-the-ones-that-play"}
listLibs: Returns an ipairs-friendly
list of libraries on the system,
such as:
{"fmttext",
"braille"}
totalIdleTime: Returns the current
kernel idle time total, useful for
measuring current CPU usage, and in
turn comparing to application CPU
time to get various statistics.
ensurePath: (s, root)
Attempts to verify the
safety of a path, and errors if any
aspect seems incorrect.
The root must be a prefix to the
path, and the path must follow a
strict standardized form that is
guaranteed to always be supported
and handled in the same way on any
OC system.
Essentially, "//" must not occur,
and all "[^/]+" matches must be
valid path components.
ensurePathComponent: (s)
Ensures that a string is a safe
filename via a character list and
some special filename checks.
UTF-8 characters are just flat out
disallowed until someone can give
me proof they won't blow up
something somewhere.
(This restriction is Windows's
fault - I can't trust the encoding
mess to not find some new and
imaginative way of breaking
filenames, so I'd rather that they
get avoided until someone can try
actually using them.)
This does NOT ACCOUNT for *all* the
Windows total nonsense (aux, com1)
because if OC doesn't cover up
that then you're kinda doomed.
ensureType: (v, ts)
Checks that a value is of a given
type, and errors otherwise. If the
type is "table", it also errors if
a metatable exists.
The additional things available to
processes are those things that
require a process to use:
pid: A field that specifies the
process ID of this process.
Harmless, but not entirely useful.
dead: Actually a field, that isn't
set at first, but is set later to
indicate deadness. Useful if your
process does anything that might
lead to functions being called in
the afterlife, such as providing an
API.
executeAsync: Function that takes
an app name (aka: pkg), and a
set of arguments to give it.
NOTE: sys- apps cannot be started
from non sys- apps no matter how
hard you try, without k.root
alterations to runProgramPolicy.
Your process pkg and ID is
prepended to the arguments.
NOTE: This uses the result, err
return format, except for security
errors in which case it uses a
full error, because you might just
ignore the return value.
A successful result is the PID.
executeExt: Like executeAsync, but
firstly, synchronous, and secondly,
with an extra first parameter that
contains a function to call on
events encountered during the time.
As for the return values, it tries
to emulate os.execute, so it
returns -1 & reason on load error,
and 0 & death-reason otherwise.
execute: executeExt, but with the
first parameter set to a blank
function.
requestAccessAsync: A function that
takes an access ID (aka 'perm') as
a string (see kn-perms for info),
and starts a security request that
is responded to with a
k.securityresponse such as:
"k.securityresponse", perm, obj
requestAccess: A function with
(perm, handler) as the arguments -
runs requestAccessAsync, then sends
events to handler (if any) while
waiting for the response.
requireAccess: requestAccess, but
(perm, reason) - the reason is used
in an error if the access cannot
be gained.
scheduleTimer: Given an os.uptime
value, creates a timer and returns
a completely meaningless table that
is never touched by the kernel
directly, called the "tag".
The resulting event:
"k.timer", tag, time, ofs
These events are ONLY EVER sent as
a consequence of this function,
and this can be relied on safely.
NOTE: Setting timers too far in the
future has effects on system
stability. So does using memory,
and there's no way for me to stop
that, either. So long as the timer
is reached, alive or dead, things
will work, but spamming timers has
the consequence of memory use,
and timers stick around after the
process that owns them is dead.
With that, I hope I have documented
the kernel's interface to programs.

70
repository/docs/kn-sched Normal file
View File

@ -0,0 +1,70 @@
This is an overview of what a program
can expect from the scheduler.
The kernel's scheduling is entirely,
and I mean entirely, timer-based.
Everything in the kernel,
that has to occur the next time the
CPU has reached the main loop,
is, without exception, a timer,
apart from the timer system itself.
That last note is important, since as
the timer system controls sleeping,
it must use computer.pullSignal -
thus, that part of the mechanism is
not in itself a timer - it is the
mechanism that waits for timers.
Signals that have been retrieved with
computer.pullSignal, however, do
become timers.
Timers are kept in a list, and have
their "target uptime" - the
computer.uptime() at which they are
due to be executed, their callback,
and after the callback, a list of
arguments to give to the callback.
The current time as KittenOS NEO
sees it is available as os.uptime().
(and the address, as os.address() -
bit of a cheat, but who's counting?)
This source is always in seconds, and
so KittenOS NEO timing is always in
seconds.
The scheduling loop's precise details
are in the kernel itself, and any
precise description would be a
translation into pseudocode of what
is already there.
But it suffices to note that the
scheduling loop works by, 16 times
at most, executing and removing all
timers from first defined to last
that have passed their time,
and getting the minimum time of all
unexecuted timers during each loop.
The last minimum time, if it exists,
is then bounded to at least 0.05,
an OC minimum value for a yield.
The pullSignal is then called with
the bounded time, if any.
(If no bounded time exists, then the
system goes into more or less a deep
freeze, which is useful to conserve
energy, even when apps are "running"
but aren't using timers.)
If there is any signal, distEvent is
called to distribute it to those
processes with the right accessses,
with an "h." prefix.

1
repository/docs/ul-broil Normal file
View File

@ -0,0 +1 @@
Hello World.

1
repository/docs/ul-neoux Normal file
View File

@ -0,0 +1 @@
Hello World.

1
repository/docs/us-perms Normal file
View File

@ -0,0 +1 @@
Hello World.