dzavalishin/phantomuserland

Phantom: Persistent Operating System

Welcome to Phantom OS

Phantom OS is a persistent operating system. Its primary goal is to provide environment for programs thatsurvive OS reboot. Such an environment greatly simplifies software development and makes it possible to write programs that for example do not need a filesystem.

Nevertheless, Phantom supports filesystems and all of the modern OS features.

For the details please refer to the Phantom OS Programmer's guide.

For the questions and discussion please visit chat or Google Group.

What Phantom OS is

• Phantom OS Programmer's guide
• Phantom Architecture in English
• Short article in English - TheRegister
• Big article in Russian - Open Systems Magazine

Basically, most of code we (the original team) upload is very straighforward and, sometimes, dumb. That, to some extent, is on purpose. We want to have a working system first and polish it next. Besides, not all the concepts and design desicions are final, so it is of no use to finalize all the implementation desisions now as well.

Code is known to compile successfully with cygwin/linux gcc 4.3.4. Only the IA32 target is most complete and stable, the ARM port is in active development but very unstable, the MIPS port has just been started - compiles and can breathe for a second :), the amd64 port is incomplete and does not compile at all.

Current state

See the changelog, have a look at screenshots. Here is the last one:

More screenshots: https://github.com/dzavalishin/phantomuserland/wiki/ScreenShots

Docs

Web Documentation and PDF, and there's a lot of info in the Wiki

Building

Set the PHANTOM_HOME environment variable to the path of the Phantom repository root directory and make all there.

On Windows you will need Cygwin to do that. Select at least: gcc4, subversion, binutils, make, gdb (see etc/cygwin_get.cmd)

See also TOOLCHAIN

Running

Run phantom.cmd/phantom.sh in /run

See doc/RUNNING for more details

Debugging

Run QEMU (see above) and then - gdb in /oldtree/kernel/phantom

Kernel console is logged to /run/serial0.log

Kernel is able to send logging info to syslogd by UDP. Currently syslogd address is hardcoded in net_misc.c.

Directories

• oldtree/kernel/phantom - kernel
• phantom - libs and unix userland (user/apps)
• plib/sys/src - native phantom userland code
• run - QEMU run/test environment
• tools/plc - phantom language compiler / java bytecode translator

Go on and take part!

How to take part

The kernel and the compiler

Creating an unusual operating system is a very interesting thing to do. There are challenges on each and every step. Just to start with:

• Persistent memory garbage collector. Suppose we're in a 64 bit world and persistent memory size is some 20 Tb. The current GC is incomplete.
• If we touch memory too much, the snapshot engine will spend a lot of IO to store the difference. A fast and good allocator can reduce IO load. There is one, but it could be better.
• There's need for a fast alpha-blending image transfer (bitblt) code.
• The Unix subsystem is very limited. There is no signal delivery, for example. It waits for the one who will implement the missing parts.
• It is theoretically possible to implement a persistent Unix environment. Quite challenging.
• Drivers - current set is minimal, The AHCI driver is not complete, USB needs optimization, some more must be ported or written.
• It would be interesting to add a Python frontend to the Phantom compiler. Are you a Python fan? Can help?
• Phantom bytecode supports classes, inheritance, but does not support interfaces. It is not really trivial to implement interfaces in an efficient way.
• Even a simple JIT engine will help a lot.
• The TCP stack is not ideal and needs someone to lend a hand.

Porting Phantom

Ports to ARM and MIPS were started, but long time no progress. I'm looking for one who can help with that.

Bringing it to 64 bit Intel/AMD is actual task too.

Current kernel is basically SMP ready, but SMP support is not finished completely.

Userland

There's a need to implement demo applications for Phantom - even simple ones will help.

More serious task is to bring in some simple HTML renderer and get a basic browser working.

Build and CI

• There is a need for a good CI setup which can run system in a specific configurations and following special scenarios.
• Bytecode engine needs to be tested with random garbage execution.
• It is a good idea to keep a set of tools that for sure build a correct OS kernel. cc, binutils, qemu, etc.
• Need setup to build an ISO image of the system to run on different machines and emulators.
• There is a real need to do CI on a real hadrware. Need corresponding scripts.

If you feel interested to take part in the project, please leave me a note. An issue on a GitHub is the ideal communications channel.

Issues to start with

Badges

Communications

The easiest way is to Write AN e-mail to Dmitry Zavalishin

Or leave a message in the Google group

Or leave an issue in the tracker

There is a Phantom Website, but since you're here you better look at the Wiki.

---

Best regards, Dmitry Zavalishin, [email protected]

phantomuserland's Issues

DEMO: video

Need demo which plays video and survives reboot, resuming play where was killed

Original issue reported on code.google.com by [email protected] on 15 Jun 2010 at 12:25

Need kernel profiling

See http://www.informatik.uni-hamburg.de/RZ/software/gnu/gcc/gprof_9.html

Original issue reported on code.google.com by [email protected] on 10 Jun 2010 at 6:26

ABSFROMROOT in stacks

Check func call code, stacks code and abs stack access instructions against
specs and real usage.

Original issue reported on code.google.com by [email protected] on 10 Aug 2009 at 1:04

atomic refcount

refcount is not proteced with spinlock and is not atomic


redo so that all negative values are forbidden?

refc++; if( refc < 0 ) refc--;

and

refc--; if( refc < 0 ) refc++;

these forms can work without locks at all - is it?

check near zero behaviour?

Original issue reported on code.google.com by [email protected] on 13 Aug 2009 at 9:27

ostack/estack references are not followed by refcount stack freeing code

all references in not-popped stack part must be followed by ref_fec_o

Original issue reported on code.google.com by [email protected] on 3 Sep 2009 at 1:56

POP_ARG doesnt decrement refcount

In fact it doesn't have to - someone who consumes the arg must!

Original issue reported on code.google.com by [email protected] on 31 Aug 2009 at 5:13

Need p2v/v2p tests

especiall for page cross situations

Original issue reported on code.google.com by [email protected] on 14 Jun 2010 at 10:47

Turn on and test MULTIPLE_DPC_THREADS

See dpc.c - MULTIPLE_DPC_THREADS is needed because we now sleep in
DPC-called code.

Original issue reported on code.google.com by [email protected] on 31 May 2010 at 5:53

Out of mem situation in physalloc

1. Must be able to decide, whether we can put caller asleep - make two
different calls for regular and interrupt usage? Or check global in
interrupt flag?

2. Enforce some memory reclaim in pager, and retry

Original issue reported on code.google.com by [email protected] on 31 May 2010 at 6:04

Reflections!

We need type/name info in phantom class files, so that reflections can be
implemented in VM.

Original issue reported on code.google.com by [email protected] on 28 Oct 2009 at 7:55

check in interrupt status in thread switch

just like we check spinlock count

Original issue reported on code.google.com by [email protected] on 14 Jun 2010 at 6:26

allocator dies if min fragment sie is 8

#define PVM_MIN_FRAGMENT_SIZE 8, run

Original issue reported on code.google.com by [email protected] on 3 Sep 2009 at 2:02

(userland) class loader execution

Class loader is started from kernel and locks up everything (namely -
snaps) while is working.

One possible solution is to replace VM command 'summon class' with direct
access to userland class loader - but then what we have to do if kernel
itself will need some class?

Other possible hack is:

- after 'summon class' is called and class is not found in fast kernel hash
(which we need to implement as well) user code IP is stepped back to the
beginning of 'summon class' instruction and thread is put asleep. After the
finish of userland class loader thread is awaken and restarts its class
request, wchich is finished quickly this time, because requested class is
already in the fast kernel hash (which...;).

Original issue reported on code.google.com by [email protected] on 5 Sep 2009 at 10:34

si_bootstrap_21_sleep races

Will loose wakeup on short sleep time.

Original issue reported on code.google.com by [email protected] on 18 Jan 2010 at 3:48

plc must generate .d files

to simplify makefiles creation

Original issue reported on code.google.com by [email protected] on 5 Sep 2009 at 10:45

Low mem BG artefact

Run OS on 80 Mb with mem reclaim turned on. Screen background will have
black hole in the middle.

Original issue reported on code.google.com by [email protected] on 3 Jun 2010 at 4:21

ATA driver does not work in BM (DMA) mode on real HW

Supposedly because of inadequate PCI config parsing: driver itself is quite
generic, and PCI config looks for very specific vendor/model instead of class.

Original issue reported on code.google.com by [email protected] on 30 May 2010 at 10:16

kernel time is converted to struct tm incorrectly

see time.c, localtime_helper()

Original issue reported on code.google.com by [email protected] on 8 Nov 2009 at 9:38

Test: amap

Need test suit for amap code

Original issue reported on code.google.com by [email protected] on 1 Jun 2010 at 7:43

phantom_bios_int_10() is not reentrant, replace

See vm86.c

Original issue reported on code.google.com by [email protected] on 10 Feb 2010 at 8:01

timed cond wait - timeout not reset when awaken manually?

call errno_t hal_cond_timedwait( hal_cond_t *c, hal_mutex_t *m, long
msecTimeout ) with long timeout

call errno_t hal_cond_signal(hal_cond_t *c)

supposedly kernel will die due to timeout awake will come to awaken thread



the same in (not ready yet) sema code?

Original issue reported on code.google.com by [email protected] on 1 Jun 2010 at 9:05

si_bootstrap_20_set_screen_background is wrong, see comments

just paints. must store.

Original issue reported on code.google.com by [email protected] on 1 Sep 2009 at 8:21

Support gcc stack check feature?

http://www.liveinternet.ru/users/whitewin/post109385474/


Need per-thread segment pointed by GS: to store temp stuff. There are a lot
of other possible uses for such segment, btw. Such as current thread
object's data area copy and fast access fields for bytecode interpreter.


Here goes an idea to use separate segment register to do object stack
access so that segment access failure will be used to extend stack.

Original issue reported on code.google.com by [email protected] on 7 Jun 2010 at 11:07

Class loader ignores base class

vm/load_class.c

Original issue reported on code.google.com by [email protected] on 25 Jan 2010 at 10:42

Free old snap page

near

pagelist_write_seq( snap_saver, p->make_page);

we can (can we?) decide if old snap's page is to be freed. Of course, it
can be freed only after writing new snapshot AND making sure it is really
written.

Original issue reported on code.google.com by [email protected] on 31 May 2010 at 5:57

Constructor code generation is disabled.

See
http://code.google.com/p/phantomuserland/source/browse/branches/j2ptranslator/pl
c/src/ru/dz/jpc/tophantom/CFile.java?spec=svn172&r=172

Constructor code generation is disabled. Fix and enable back.

Original issue reported on code.google.com by [email protected] on 30 Oct 2009 at 9:34

pvm_exec_throw must do VM-level throw

need longjmp there in exec.c?

Original issue reported on code.google.com by [email protected] on 10 Aug 2009 at 1:14

VESA farmebuffer mapping

See map_video() in video_drv_bios_vesa.c - upper variant maps videobuffer
directly, and it works OK. Lower one maps it to allocated address space,
and it fails (hardware reboot, not even panic).

Original issue reported on code.google.com by [email protected] on 28 Oct 2009 at 8:04

window hor scroll

make syscall for existing code

Original issue reported on code.google.com by [email protected] on 14 Jun 2010 at 10:53

TODO: ACPI BIOS interface

todo

Original issue reported on code.google.com by [email protected] on 27 Jan 2010 at 2:56

amap test fails

Run krenel tests. Result is:

-----
Test fail: modified is not true
!!! KERNEL TEST FAILED: amap -> -1
-----

relevant code is (in test_amap.c)

    test_check_false(  amap_check_modify( &map, 10, 1, MAP_USED, &modified ) );
    test_check_false(  amap_check_modify( &map, 10, 1, MAP_USED, &modified ) );
    test_check_false( modified );

    test_check_false(  amap_check_modify( &map, 10, 1, MAP_FREE, &modified ) );
    test_check_true( modified );

Failing function is 'amap_check_modify', which must set &modified to true
when modifying map entry.

Original issue reported on code.google.com by [email protected] on 5 Jun 2010 at 11:43

allocator dies on second attempt to alloc 400K object

just run pvm_test

Original issue reported on code.google.com by [email protected] on 3 Sep 2009 at 1:57

Ne2000 interrupts

check other drivers for polling as well

Original issue reported on code.google.com by [email protected] on 14 Jun 2010 at 11:31

PHANTOM_OBJECT_STORAGE_FLAG_IS_CALL_FRAME is not set on callframe creation

for now hacked in in object init code, which is wrong

Original issue reported on code.google.com by [email protected] on 3 Sep 2009 at 1:55

Need memory mapped binary class

Mapped binary class can be used to map into the OS address space external
data, such as legacy FS file, etc

Original issue reported on code.google.com by [email protected] on 21 Sep 2009 at 8:10

ISA devices duplicates

driver/probe_isa: Driver 'LPT1' attached to ISA at 0x378 (IRQ 7)
com/com_detect: com1: 82550 or 16550 chip
driver/probe_isa: Driver 'COM1' attached to ISA at 0x3F8 (IRQ 4)
pager/vm_map_deferred_disk_alloc_thread: Ready
pager/vm_map_lazy_pageout_thread: Ready
boot/main: Will run phantom threads... 
driver/phantom_pci_find_drivers: Look for PCI devices, stage 4
driver/probe_isa: Driver 'LPT1' attached to ISA at 0x378 (IRQ 7)
com/com_detect: com2: 82550 or 16550 chip
driver/probe_isa: Driver 'COM1' attached to ISA at 0x3F8 (IRQ 4)

Original issue reported on code.google.com by [email protected] on 6 Jun 2010 at 11:13

Bogus physram

boot/dumpf: [0x0049B21F5 - 0x0080F0000[ (  55 M) - hi RAM

In fact 0x008000000 to 0x0080F0000 seems to be non-working RAM.

Why?

Original issue reported on code.google.com by [email protected] on 14 Jun 2010 at 11:29

Semaphores

Need semas such that sem_release can be called from interrupt

Original issue reported on code.google.com by [email protected] on 14 Jun 2010 at 10:50

Object land access to hostname

see unix/sys_net.c

Original issue reported on code.google.com by [email protected] on 15 Feb 2010 at 1:09

need timed call reset (remove from list!)

way to bring timedcall struct out of processing - btw, races!

Original issue reported on code.google.com by [email protected] on 1 Jun 2010 at 9:07

si_array_11_set refcount increment missing

added     ref_inc_o(value);

Original issue reported on code.google.com by [email protected] on 31 Aug 2009 at 9:02

DPC tests

Test to check DPC, incl multithreaded mode

Original issue reported on code.google.com by [email protected] on 1 Jun 2010 at 9:08

Need hashmap class w complete access from C

for things like maps of settings, etc

Original issue reported on code.google.com by [email protected] on 4 Jun 2010 at 9:26

Pageout and mem reclaim

- lazy pageout
- queue of pages to be ready to take mem off
- some pre-cleaned page queue?

Original issue reported on code.google.com by [email protected] on 31 May 2010 at 6:07

IDE driver - DMA mode does not work on real hardware

1. Enable DMA in paging_device.c
2. Run OS on real computer
3. panic.

Need setup network logging to see exact panic cause.

Original issue reported on code.google.com by [email protected] on 11 May 2010 at 9:40

• Merged into: #25

mutexes in internal objects

how do they cross snapshots? weakref is good example

Original issue reported on code.google.com by [email protected] on 12 Jun 2010 at 8:44

Need partition handling / new disk io stack tests

via trfs?

Original issue reported on code.google.com by [email protected] on 14 Jun 2010 at 10:48

spinlocks can't be used in weakrefs

See KernelSync notice on spinlocks and access to persistent land.
See comments in create.s to pvm_weakref_get_object

Original issue reported on code.google.com by [email protected] on 12 Jun 2010 at 8:21

IP stack - change retcodes to errno_t

Change constants as well!

(or just redefine newos consts with errno ones?)

Original issue reported on code.google.com by [email protected] on 1 Jun 2010 at 7:28

Aligned object data

To implement effectively large-scale data strucrtures I believe it is
needed to have:

1. Some way to allocate binary and array object with total (including any
allocation headers) size equal to the system's default page size and aligned
on page boundary.

2. Some way to allocate binary object with CONTENTS aligned on page
boundary and size which is multiplication of page size. (This one is for
native code exec)

Original issue reported on code.google.com by [email protected] on 31 Aug 2009 at 5:12