It was custom. It booted from ROM straight into BASIC (which was written by Microsoft as just about every BASIC was). To load most commercial games or other software you would insert a disk (the disk drive was a separate 5.25" device) and type 'LOAD"*",8,1'

Since disks could contain multiple programs you could get a directory by typing 'LOAD"$",8' and then 'LIST'

Then you could load the program by typing 'LOAD"PROGRAM NAME",8,1' and then 'RUN'

DOS was built into the disk drive which had its own CPU but you only typically interacted with it through the sort of cryptic 'LOAD' and 'SAVE' commands from within BASIC. It wasn't something you booted into.

Games were roughly the quality of original Nintendo games. The Commodore 64 was superior in some ways in that regard and inferior in others.

The Commodore 64 mostly competed with the Atari 8-bit computers and the Apple II line.