Nice NetBSD/dreamcast shell script, but can you do it under Windows?

I posted about the dc-burn-netbsd script on one of the Dreamcast forums, and someone asked if they could run the script under Windows.

I suspect pointing him at Microsoft's current *nix-on-Windows solution (Interix? Windows Services for Unix?  Subsystem for UNIX-based Applications? Grudgingly Provided but Subtly Maladapted Compatibility Subsystem for Customers Who Otherwise Would Use Some OS Begining with L?)... might have (quite rightly) have been met with a vigorous but somewhat impolite reply, so I decided to generate an image for him to burn directly.

I also took the opportunity to cleanup dc-burn-netbsd a little more:

• split out 'extract sets' from 'setup extracted data for live usage' (now -l) - after all, why should I assume my live setup is the One True Way?
• tweaked live setup further (do we *really* want to try to rebuilding the X11 fontcache? I think not), and /etc/motd was not reporting the kernel version (tsk, fer-shame)
• making -l default to the right kernel and minimal sets. I have a deep seated antipathy towards typing many options on a command line. (If I'm building a live CD then it should be able to work out that it needs some sets and a non ramdisk kernel - but only as a default of course, if the user wants to explicitly select something apparently stupid... "here is rope")
• Adjusting default behaviour to generate but not burn the image (added -b to burn). Ah, now I have a naming crisis as it no longer "does what it says on the tin" by default. Foo. I'll come back to that later

He also asked another question - to paraphrase it - "why?"

Currently the motivation for all this has been "because its there" - while NetBSD/dreamcast can run a whole bunch of standard unix software (see http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/dreamcast/5.1_2012Q1/All/ for example), not all of it necessarily makes sense on such a platform.

But again... "here is rope" :)



I also put together a README (I'll quote it below - mainly on the grounds I've already spent time writing it, got distracted watching the Olympics opening ceremony, and its a cheap way to pad out the rest of this entry)


netbsd-dreamcast-6_BETA2.iso.README

Sample NetBSD/dreamcast live image (Generated by "dc-burn-netbsd -l -n"):

Requirements
- Dreamcast or gxemul emulator (*)
- If using a real Dreamcast, blank CD-R & Dreamcast keyboard
- Optional: Dreamcast BBA (broadband adaptor) for network access

Usage:
- Download netbsd-dreamcast-6_BETA2.iso.7z and uncompress to get .iso image
- To burn to CD for booting on Dreamcast:
  - On Windows use http://code.google.com/p/bootdreams/downloads
  - On NetBSD no need to download this, use http://pkgsrc.se/sysutils/dc-tools
    and run 'dc-burn-netbsd -l' directly
- To boot in gxemul (*) run
  "gxemul -XEdreamcast -d co23965696:netbsd-dreamcast-6_BETA2.iso"
- Once booted enter "gdrom0" and return three times when prompted
- Login as "root"

This is still a work in progess, in particular future goals for future versions
include:
- Including installed binary packages such as some from
  http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/dreamcast/5.1_2012Q1/All/
- Switching to a shell with command line editing
- Support for Serial-SD and potentially Serial-Ethernet adaptors
- Better X support (currently insufficient memory to effectively run)

(*) Running under gxemul requires gxemul-0.6.0nb3 or later from pkgsrc, or the patch-src_devices_dev__dreamcast__gdrom.cc patch applied to gxemul-0.6.0

Interestingly X doesn't  start under gxemul - I'll need to test on real hardware before I dig further into that, but as my DC is in my office, and I'm currently in a Days Inn in Winchester (not Inglewood - that was a rather different experience), that will have to wait...

Virtual Dreamcast - Virtual (a)Live?

There has been something of a hiatus in my Retrochallenge 2012 postings.

Last Tuesday evening I visited London Hackspace and... made it home in the decidedly early hours of Wednesday morning, rather neatly occupying the time during which I would have tried to do something Retrochallenge related and then have written about it.

Then as ever, "stuff happened", but finally I have made time to get back to it!

"...Last episode on Retrochallenge2012, I was doing something with a Dreamcast, or a VAX - I really can't remember now, possibly even combining the two in a gratuitous and potentially amusing fashion, but I did remember that trying to create a Dreamcast live CD was quite an annoying process, requiring as it did burning a CD-R each time to test.

Enter gxemul, a very nice set of emulators which included one that could boot NetBSD/dreamcast, including a rather elderly NetBSD-3.0 live CD image. Unfortunately... I was unable to get a recent NetBSD CD image to mount - the kernel would load, I could view the start of the CD image partition which looked fine, but it would just refuse to mount the ISO9660 filesystem.

Sprinkling some debugging prints in both the kernel and gxemul (it is *so* much nicer to be able to debug from both sides), revealed the NetBSD kernel trying to access sectors somewhere in the multigigabyte range (a neat trick on a medium with a maxiumum of 900MB capacity).

The issue, it transpires, was gxemul only fakes up enough of a CD ToC (Table of Contents) to allow NetBSD to boot... specifically NetBSD 3.0. NetBSD 4.0 and later have stricter requirements. Actually the code is very similar and I have a horrible feeling that the fake ToC caused the NetBSD 3.0 gdrom code to fall off the start of an array and read some nulls. Nasty.

So I went searching for the CD ToC specifications, and eventually determined that I needed the Yellow and Red Book standards, written by Sony & Philips and apparently still to this day considered commercially privileged and not freely available. However... their contents match the freely available (and catchy named) Standard ECMA-130: Data Interchange on Read-only 120 mm Optical Data Disks (CD-ROM) which has insane level of detail on the ToC (among other things).

So I read the relevant section, rubbed my eyes a few times and read it again, then took out a pad & paper and thought about drawing diagrams... and then read it again. I considered the strange need for some engineers to produce overly complicated and involved solutions, and about technical writers who actively despised their potential readers.

I just have to share Figure 15 at this point. Looks nice and simple? Just split up 96 bits into sections? Only... bits 0..97 makes *98* bits, and the Control bits have two extra bits (for free). Don't even get me started on how each track has 4 bytes in the ToC data returned, but something like 10 bytes in the spec.


Then I experimented with building gxemul with some values that should have made sense... but apparently didn't. In the end I burned a NetBSD/dreamcast disk with a modified kernel which dumped the ToC values, then updated gxemul to fake a more convincing ToC...

Final result is an updated gxemul which will boot NetBSD/dreamcast 3, 4, 5 & 6 kernels, a NetBSD/dreamcast kernel option to dump the gdrom ToC, and a tweak to the gdrom parsing code to reject too-fake ToC values.

Once the above was reached it only took a few quick iterations to get dc-burn-netbsd building fully functional Dreamcast live CDs, including the ability to login to both the graphical and the serial consoles - literally multiple users on a virtual Dreamcast without any network interfaces or persistent storage, virtual or otherwise. And they say retrocomputing emulation has no practical use? Pffff...

wargames meets Linux (and Dreamcast VGA)

When I wrote a small script to simulate the W.O.P.R. computer from wargames, little did I expect it to help me learn some new things about Linux (specifically Fedora in this case).

One criticism often cast towards people writing software on Linux is that they only write it for Linux, not caring about basic portability to other systems. Sometimes there has obviously been a pass to make it run on OS/X also, though earlier versions of XBMC were my go-to case for "portability" code which only cthulhu could love (So, if #ifdef _LINUX and #ifdef APPLE then... no, wait, what?)

Anyway, My Little Pony-I-mean-Wargames script was written on NetBSD, for my own use, but not intentionally using any NetBSD specific features.

Since I had the temerity to host it on github, of course someone is going to come along and try to build it on their $OS_of_choice, in this case Fedora.

So, lets count my portability fails

1. I used "tput up" to move the cursor up, rather than "tput cuu1" (both work on NetBSD). To be fair "cuu1" does sound more like something a standards committee would define, probably short hand for "CursorUnproportionablyUpOneLineOnly"
2. I used "{.TARGET}" and "${.ALLSRC}" in the Makefile. Gmake does not support these, which is interestingly as I thought the gmake project mission statement was to make bash look like a lean, minimally functional application. Still, easy enough to hardcode the values & we're portable
3. I include a getopt(3) using program for outputting text called wopr, which at one point was called as 'wopr "IDENTIFICATION NOT RECOGNIZED BY SYSTEM" "--CONNECTION TERMINATED--" ""'. Traditional getopt() usage has option parsing stop when the first non '-' prefixed argument is reached. Linux is more... helpful... in this respect and happily picks up options from anywhere in the arguments... and quite possibly from the environment and your dotfiles. Wait... (check online), No, apparently I only *thought* I was kidding:

ENVIRONMENT VARIABLES

_<PID>_GNU_nonoption_argv_flags_

This variable was used by bash 2.0 to communicate to GNU libc which arguments are the results of wildcard expansion and so should not be considered as options. This behaviour was removed in bash version 2.01, but the support remains in GNU libc.

 So, now wargames now runs on Linux. Anyone with a Solaris/Illuminos machine want to take a pass? :)

(I'd like to thank the person on github who took the time to try wargames, and then fix it up to run on Fedora, and who should not take anything of the above as ingratitude :)

... meanwhile, on planet Dreamcast

In an earlier posting I may have commented on the quality of the construction of a newly acquired Dreamcast SD+VGA adaptor in a fashion which might have been construed as potentially disparaging. I am now delighted to revise that previous comment and to be able to state positively that indeed, it is a bit sh*t. By repeatedly pressing the plug into the back of the Dreamcast, and - god help me yes - by tilting to engage the assistance of gravity, I now have a clear and fully usable display. As before, walking heavily, sneezing, or careless placing of coffee cups near the machine is strictly prohibited.

I appear to be getting carried away in mentioning previous postings, in this case why gxemul was not able to mount the CD as a root filesystem while booted. I am lightly disappointed to say it does not appear to be some boneheaded mistake in my script as the CD image runs fine on real hardware, so if I want to use gxemul to test root-on-cd9660 Dreamcast image I'm going to have to poke into the gxemul source. Definitely still retrochallenge 2012 material, though only just :)

Tidying up at the end of the week

Spot the server, printer, or cow

Spent some of today tidying up the house, and a little bit tidying up in the Dreamcast and VAX emulation world.

• Updated the Booting NetBSD/vax in SIMH howto to reflect the changes in NetBSD-6, point to the correct page for the simh pkgsrc package and misc cleanup
• Updated the NetBSD/dreamcast howto to point at the two very much easier methods of burning a Dreamcast CD
• Relaxed the timer granularity check on the SIMH pkgsrc package to allow setting the idle check on machines with HZ=100 - now emulated vax machines do not need to max out the host CPU all the time
• Fixed the configure mkstemp() check on the gxemul pkgsrc package, so it can boot full Dreamcast CD images again
• Adjusted the dc-burn-netbsd script to include a full set of devices in the release-in-root-on-cd case, and attempted to test this in gxemul (hence the previous item)

Interestingly I discovered that once the CD image was booted in gxemul attempting to mount the CD failed with the (verbose) gxemul console messages:

[ GDROM cmd: 30 20 39 fd 9c 00 00 00 00 00 01 00 (cnt=2048) ]
[ diskimage__internal_access(): disk_id 0, offset 2791911424, transfer not completed. len=2048, len_done=0 ]
GDROM: diskimage_access failed? TODO 

Given the entire ISO image was 97,910,784 bytes in size, I suspect tying to access data at offset 2,791,911,424 is unlikely to yield a happy result in any universe with congruent natural laws to our own, but just to make sure I'll try to test on a real Dreamcast tomorrow.

Looks like I'll either be hacking my dc-burn-netbsd script to determine what boneheaded bug I've added, or gxemul's dev_dreamcast_gdrom.cc. Good good!

Potential VAX gcc news (and damp dogs)

Today was not particularly productive in Retrochallenge terms - I visiting my parents and spent a large part of it digging fresh palisade roll into their garden, ably "assisted" by two dogs, one of whom tried to stand exactly wherever I needed to be, and the other just dug up random sections of flowerbed.


Fortunately it was raining continually, which should have driven the dogs indoors, rather less fortunately they were so caught up in helping me that they remained outside the entire time, finally following me back in trailing that distinctive fragrance of "damp dog"... and mud. Don't forget the mud.

I did manage to exchange email with someone who is both an expert on VAX assembler and gcc, who now has an simh NetBSD environment setup, and is investigating the possibility of fixing the latest gcc to be able to compile VAX executables which bear at least a passing resemblance to their source code.

This is definitely a Good Thing (in VAX terms), and would benefit anyone looking to compile VAX executables for any system using a recent gcc (Granted, this may be a rather exclusive club).

Oh, and I did fix a recently added reference in the NetBSD atf tree which unconditionally pointed to the gcc-4.5 stdc++, breaking the gcc-4.1 build (and I'm going to go out on a limb and guess the set of people who care about that make the group in the previous paragraph seem like a thundering herd).

Dreamcast display fail. Now with added SD-card!

When I first connected up my Dreamcast to boot NetBSD I used a generic RCA video to VGA convertor. It cames with its own little power adaptor and the sort of multi-button UI designed by monitor engineers from the late 90's who had clearly been through a messy divorce and wanted to share those feelings with the rest of the world.

The display was faithful to what you would see on a normal video monitor - so moderately awful - and faithfully reproduced the "we'll just hide the top or bottom two lines of your console to make typing a more challenging experience" effect. Starting X was particularly visually interesting, with the root weave pattern providing a crawling sensation on the insides of your eyeballs.

When I found out that Dreamcast SD card adaptors were easily available I immediately set out to buy one, but was then tempted by a combined VGA+SD, with its promise of crisper display, and the sensual delight of potentially being able to characters on screen as you typed them.

It arrived today, and was hooked up to the DC when I arrived in the office. Build quality was... adequate for something that had been assembled from parts left lying around in a workshop. In particular the two plugs appear to have been designed by someone who once spoke to someone who had the original plug described to them over a phone... in a foreign language. They fit, but much in the same way as breaking out all of the pins in a serial cable, then individually poking them into the serial port would fit. For a moment I seriously considered turning the Dreamcast onto its front edge to gain the assistance of gravity with keeping the plugs in, but in the end resolved just not to sneeze or walk heavily near it.

As for the display... the good news is that all of the lines from the console are visible on screen, the bad news is you need to take quite a relaxed view of the word "visible" . Think "ancient TV with something significant dying inside", with a side order of "who turned the brightness all the way down and broke off the knob".

Its a little disappointing to say the least. I will have another play on Monday then quite possibly email some pictures accompanied by a light sprinkling of sarcasm to the vendor...

NetBSD/dreamcast from NetBSD in 4 minutes?

So last night I located an easy way to burn a NetBSD/dreamcast bootable CD from a Windows box, along the way discovering it was *almost* possible to use that mechanism to have the iso9660 filesystem used as the root fs.

It annoyed me that it was now quicker to burn a NetBSD/dreamcast CD on a Windows box than a *ix box, so tonight I decided to address that.

A fair number of years ago Marcus Comstedt write a number of very handy Dreamcast development tools, including a couple of utilities to generate a Dreamcast bootloader and then process a binary suitable for use with that bootloader ('makeip' and 'scramble' from http://mc.pp.se/dc/sw.html)

All I needed to do was wrap them in a little shell script, throw in a call to mkisofs to create an iso9660 CD filesystem and cdrecord to write and and I was done.

Then the creatures started feeping.

• Obviously it would be convenient if the script could download the necessary kernel files from ftp.netbsd.org automatically, but you should also be able to point it at a locally built kernel
• If downloading the version should be selectable
• Selection of a plain GENERIC or GENERIC plus ramdisk be possible
• As a final bonus it should be able to download a complete NetBSD distribution, extract and burn it onto the CD as the root filesystem

The end result was dc-burn-netbsd on github, and a sysutils/dc-tools pkgsrc entry which included dc-burn-netbsd, makeip, scramble, and a couple more dc tools.

... and finally the feeping stopped.

I can now say I've run a Dreamcast with an iso9660 (rockridge) root filesystem. Its not exactly fast, and you would really want to union mount some ramdisks over /var and suchlike to even consider trying to take it multiuser, but it works :)


% dc-burn-netbsd -h
Usage: dc-burn-netbsd [opts]
-C : Clean work directory before starting
-c opts : Set cdrecord opts (driveropts=burnfree gracetime=3)
-d : Take kernel/ & sets/ under datadir - no downloading
-h : This help
-k type : Set kernel type (GENERIC_MD) eg: GENERIC or GENERIC_MD
-n : Generate data but do not write (just display cdrecord commands)
-r : Include full NetBSD release on CD
-t tmpd : Set temporary work directory to tmpd
-v vers : Set NetBSD version (6.0_BETA2) eg: 5.1 6.0_BETA2

dc-burn will create a temporary work directory dc-burn-netbsd-files which will
need to have sufficient space to store the downloaded & generated files.

if -d is used the directory is expected to match the layout on ftp.netbsd.org:
- kernel/netbsd-$type.bin.gz, and
- sets/base.tgz (etc - if -r given)