Previous Thread
Next Thread
Print Thread
Page 2 of 3 1 2 3
Re: Compile stats [Re: Golden Child] #112303
01/23/18 10:25 PM
01/23/18 10:25 PM
Joined: Jul 2007
Posts: 209
California
M
Monotremata Offline
Senior Member
Monotremata  Offline
Senior Member
M
Joined: Jul 2007
Posts: 209
California
Ahh gotcha! Other than running Ultima its hard to tell whats he's doing on the Apple before firing it up.

Re: Compile stats [Re: Golden Child] #112495
02/05/18 11:59 AM
02/05/18 11:59 AM
Joined: Feb 2014
Posts: 146
G
Golden Child Offline OP
Senior Member
Golden Child  Offline OP
Senior Member
G
Joined: Feb 2014
Posts: 146
Some more silly compiling fun...

I am so enamored with lua that I thought, why not install ubuntu's lua so I can write lua scripts outside of mame.

After a "sudo apt-get install lua5.3" I can run a lua interpreter from a regular bash shell.

One of the things I like is to see what the progress is for a compile.

So why don't we pipe the output from a make to a lua script and add the seconds for fun:

This is a really simple one liner. We just get the current time and then read a line, and check the time.

Then we print the time since the compile started and then the line that was read.

Let's call it "lua_time.lua"
Code
l = os.time() while true do j = io.read() k = os.time() if j == nil then break else print(k-l, j) end end


You can try this code in the lua interpreter by launching the interpreter with "lua5.3" and then pasting the line.

Type a few test lines to see what happens.

Code
lua5.3
Lua 5.3.3  Copyright (C) 1994-2016 Lua.org, PUC-Rio
> l = os.time() while true do j = io.read() k = os.time() if j == nil then break else print(k-l, j) end end
hey
3	hey
this
5	this
works
9	works
> 


I press CTRL+D to signal the end of input and the while loop drops out since it reads nil.

Another CTRL+D will drop out of the lua interpreter.

Ok, so now we've seen that our one liner works. So let's see it in action:

Our command will be: "make -j4 | lua5.3 lua_time.lua | tee luatimemake.txt"


Code
~/Downloads/mame194_extract/mame$ make clean
GCC 7 detected
Cleaning genie
Cleaning...
make[1]: Entering directory '/home/customer/Downloads/mame194_extract/mame/src/devices/cpu/m68000'
Cleaning...
make[1]: Leaving directory '/home/customer/Downloads/mame194_extract/mame/src/devices/cpu/m68000'


~/Downloads/mame194_extract/mame$ make -j4 | lua5.3 lua_time.lua | tee luatimemake2.txt
0	GCC 7 detected
0	Creating obj/Release
0	Creating obj/Release/src/host
0	Creating obj/Release/src/host/lua-5.3.0/src
0	lapi.c
1	lauxlib.c
1	lbaselib.c
1	Converting src/mame/drivers/ymmu100.ppm...
2	make[1]: Entering directory '/home/customer/Downloads/mame194_extract/mame/src/devices/cpu/m68000'
4	lbitlib.c
4	lcode.c
4	lcorolib.c
4	Linking m68kmake...
5	lctype.c
5	ldblib.c
...etc


So after a run, we've got a file that has a log of the compile run with a time at the beginning of each line.

Let's write a little routine that opens that file, and reads it into a table where the keys are each line of the compiler output and the value is a table with two entries, the line number of the output and the time since the beginning of the compile:

Let's call this file: "lua_checktime_simple.lua".

We just match the line with the pattern (.+)\t(.+) which gets us two items separated by a tab character, and these get returned as a table since we wrap it in braces {}. We read a line into g, then set h = {"g:match("(.+)\t(.+)")}. h[1] will be the time, h[2] will be the compiler output line.

Then we make a new table using the compiler output line as the key with t[h[2]] = {}.

t[h[2]][1] = linenum sets the first entry to the current line number
t[h[2]][2] = h[1] sets the second entry to the time that we got from the match.

Code
function formattime(a) local min,sec = math.modf(a/60) return min..":"..string.format("%02.0f",sec*60) end

f = io.open("luatimemake.txt","rb")

linenum = 0
t = {} f:seek("set") g = f:read() while g ~= nil do h = {g:match("(.+)\t(.+)")} maxval = tonumber(h[1]) linenum = linenum + 1 if h[2] ~= nil then t[h[2]]={} t[h[2]][1]=linenum t[h[2]][2]=tonumber(h[1]) end g = f:read() end

itemcount = 0
for a,b in pairs(t) do itemcount = itemcount + 1 end
print("TOTAL ITEMS:" .. itemcount)

newcount = 0


starttime = os.time()


while true do

newline = io.read()
newcount = newcount + 1
if newline == nil then break end
newlinetime = os.time() - starttime
if t[newline] ~= nil then oldlinetime = t[newline][2] else oldlinetime = nil end

print (newline)

if oldlinetime == nil then print("*** line not found *** NEW >>> "..newline) else print("OLDTIME = " .. formattime(oldlinetime).." "..oldlinetime .. " NEWTIME = " .. formattime(newlinetime).." ".. newlinetime .. " COMPARISON = " .. string.format("%2.1f",(oldlinetime / newlinetime)*100) .."% ".. " ITEMCOUNT = " .. newcount .."/"..itemcount) end
end


And if we invoke this with: "make -j4 | lua5.3 lua_checktime_simple.lua"

Code
~/Downloads/mame194_extract/mame$ make -j4 | lua5.3 lua_checktime_simple.lua 
TOTAL ITEMS:8514
GCC 7 detected
OLDTIME = 0:00 0 NEWTIME = 0:00 0 COMPARISON = -nan%  ITEMCOUNT = 1/8514
Creating obj/Release
OLDTIME = 0:00 0 NEWTIME = 0:00 0 COMPARISON = -nan%  ITEMCOUNT = 2/8514
Creating obj/Release/src/host
OLDTIME = 0:00 0 NEWTIME = 0:00 0 COMPARISON = -nan%  ITEMCOUNT = 3/8514
Creating obj/Release/src/host/lua-5.3.0/src
OLDTIME = 0:00 0 NEWTIME = 0:00 0 COMPARISON = -nan%  ITEMCOUNT = 4/8514
lapi.c
OLDTIME = 0:00 0 NEWTIME = 0:00 0 COMPARISON = -nan%  ITEMCOUNT = 5/8514
lauxlib.c
OLDTIME = 0:00 0 NEWTIME = 0:01 1 COMPARISON = 0.0%  ITEMCOUNT = 6/8514
lbitlib.c
OLDTIME = 0:00 0 NEWTIME = 0:01 1 COMPARISON = 0.0%  ITEMCOUNT = 7/8514
lbaselib.c
OLDTIME = 0:00 0 NEWTIME = 0:01 1 COMPARISON = 0.0%  ITEMCOUNT = 8/8514
Converting src/mame/drivers/ymmu100.ppm...
OLDTIME = 0:00 0 NEWTIME = 0:01 1 COMPARISON = 0.0%  ITEMCOUNT = 9/8514
lcode.c
OLDTIME = 0:00 0 NEWTIME = 0:01 1 COMPARISON = 0.0%  ITEMCOUNT = 10/8514
lcorolib.c
OLDTIME = 0:00 0 NEWTIME = 0:01 1 COMPARISON = 0.0%  ITEMCOUNT = 11/8514
lctype.c
OLDTIME = 0:01 1 NEWTIME = 0:01 1 COMPARISON = 100.0%  ITEMCOUNT = 12/8514
ldblib.c
OLDTIME = 0:01 1 NEWTIME = 0:01 1 COMPARISON = 100.0%  ITEMCOUNT = 13/8514
ldebug.c
OLDTIME = 0:01 1 NEWTIME = 0:01 1 COMPARISON = 100.0%  ITEMCOUNT = 14/8514



Now I can load this baseline compiler log on another system and compare the cpu speed.

Just some silly fun...

Re: Compile stats [Re: Golden Child] #112579
02/11/18 05:57 PM
02/11/18 05:57 PM
Joined: Feb 2014
Posts: 146
G
Golden Child Offline OP
Senior Member
Golden Child  Offline OP
Senior Member
G
Joined: Feb 2014
Posts: 146
I wanted a progress bar when I compile so I added that, along with a "dog bark" to tell me when it's done:

Once you've made a reference file in "luatimemake.txt" you get a nice progress bar.

make -j4 | lua5.3 compile_progress.lua


Code
-- compile_progress.lua


function dogbark() for i = 1,5 do os.execute("play -q /usr/share/sounds/gnome/default/alerts/bark.ogg") end end
-- for dogbark to work you have to "sudo apt install sox"
dogbark()

function formattime(a) local min,sec = math.modf(a/60) return min..":"..string.format("%02.0f",sec*60) end

f = io.open("luatimemake.txt","rb")
fileout = io.open("luacompareoutput.txt","w")

function progressbar(size,fraction)
local numbars = math.floor((size-2)*(fraction))
local fracpart = ((size-2)*fraction)-numbars
--local onechar = string.sub("01234567890",math.floor(fracpart*10)+1,math.floor(fracpart*10)+1)
--local onechar = string.sub(".,;:-=+\\|/*#",math.floor(fracpart*10)+1,math.floor(fracpart*10)+1)
  local onechar = string.sub(".abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ",math.floor(fracpart*53)+1,math.floor(fracpart*53)+1)
local numremain = (size-2)-numbars
if numbars >= size - 2 then onechar = "" end
  return "["..string.rep("#",numbars)..onechar..string.rep(".",numremain-string.len(onechar)).."]".." "..string.format("%6.2f%%",100*fraction) --..fracpart..numbars
end

-- for i =0,100,0.5 do print (i,progressbar(80,i/100),progressbar(10,i/100)) end

function pprint(a) fileout:write(a .. "\n") print(a) end
function printt(t) for a,b in pairs(t) do print (a,b) end end

function dumpstring(a) for k = 1,string.len(a) do print (k,string.format("%3d",string.byte(a,k)).." "..string.sub(a,k,k)) end end

function add_to_table_and_check_duplicate(linestr,timeval,t,linenum)
local linestr2
if linestr ~= nil then
-- print(linestr) 
  if t[linestr]~= nil then 
    t[linestr]["count"] = t[linestr]["count"]+1  
    linestr2 = linestr .. "<DUPLICATE LINE NUM=" .. t[linestr]["count"]..">" print(linestr) 
  end
  if t[linestr]==nil then t[linestr] ={linenum=linenum,timeval=tonumber(timeval),count=1 }
                     else t[linestr2]={linenum=linenum,timeval=tonumber(timeval),count=1 }
                        pprint ("Original at "..t[linestr]["linenum"].." Duplicate at "..linenum..":\""..linestr2.."\" origtime = "..t[linestr]["timeval"].." newtime = "..timeval) 
  end 
end 
if linestr2 ~= nil then return linestr2 else return linestr end
end

linenum = 0

arun_table = {} f:seek("set") g = f:read() 
while g ~= nil do 
timeval,linestr = g:match("(.-)\t(.*)") 

maxval = tonumber(timeval) 
linenum = linenum + 1 

add_to_table_and_check_duplicate(linestr,timeval,arun_table,linenum)

g = f:read() end

aitemcount = 0
for a,b in pairs(arun_table) do aitemcount = aitemcount + 1 end
print("TOTAL ITEMS:" .. aitemcount)

newcount = 0

brun_table = {}

starttime = os.time()

while true do

newline = io.read()
newcount = newcount + 1
if newline == nil then break end

if newline:match("\t") ~= nil then
  newlinetime,newline = newline:match("(.-)\t(.*)") 
else
  newlinetime = os.time() - starttime
end

newline = add_to_table_and_check_duplicate(newline,newlinetime,brun_table,newcount)

if arun_table[newline] ~= nil then oldlinetime = arun_table[newline]["timeval"] oldlinenum = arun_table[newline]["linenum"] else oldlinetime = nil end

pprint (newline)

if oldlinetime == nil then pprint("*** line not found *** NEW >>> "..newline) print(arun_table[newline]) else pprint("REF TIME A = " .. formattime(oldlinetime).." "..string.format("%4d",oldlinetime) .. " TIME B = " .. formattime(newlinetime).." ".. string.format("%4d",newlinetime) .. " SPEED = " .. string.format("%2.1f",(oldlinetime / newlinetime)*100) .."%".. " BCOUNT = " .. string.format("%4d",newcount) .."/"..string.format("%4d",aitemcount).." ".."ALINE = " .. string.format("%4d",oldlinenum) .."/"..string.format("%4d",aitemcount).." ".. progressbar(52,oldlinenum/aitemcount)) 
   brun_table[newline]["match"]=1
   arun_table[newline]["match"]=1
end
end

print("NOMATCH IN A")
for c,d in pairs(arun_table) do if arun_table[c]["match"]==nil then print ("NOMATCH A:"..c) end end
print("NOMATCH IN B")
for c,d in pairs(brun_table) do if brun_table[c]["match"]==nil then print ("NOMATCH B:"..c) end end

print("DUPLICATES IN A")
for c,d in pairs(arun_table) do if arun_table[c]["count"]>1 then print (arun_table[c]["count"].."x:".."\""..c.."\"") end end
print("DUPLICATES IN B")
for c,d in pairs(brun_table) do if brun_table[c]["count"]>1 then print (brun_table[c]["count"].."x:".."\""..c.."\"") end end

dogbark()


and the output looks like:

Code

GCC 7 detected
REF TIME A = 0:00    0 TIME B = 0:00    0 SPEED = -nan% BCOUNT =    2/8528 ALINE =    1/8528 [..................................................]   0.01%
Creating obj/Release
REF TIME A = 0:00    0 TIME B = 0:00    0 SPEED = -nan% BCOUNT =    3/8528 ALINE =    2/8528 [..................................................]   0.02%
Creating obj/Release/src/host
REF TIME A = 0:00    0 TIME B = 0:00    0 SPEED = -nan% BCOUNT =    4/8528 ALINE =    3/8528 [..................................................]   0.04%
Creating obj/Release/src/host/lua-5.3.0/src
REF TIME A = 0:00    0 TIME B = 0:00    0 SPEED = -nan% BCOUNT =    5/8528 ALINE =    4/8528 [a.................................................]   0.05%
lapi.c
REF TIME A = 0:00    0 TIME B = 0:00    0 SPEED = -nan% BCOUNT =    6/8528 ALINE =    5/8528 [a.................................................]   0.06%
lauxlib.c
REF TIME A = 0:00    0 TIME B = 0:00    0 SPEED = -nan% BCOUNT =    7/8528 ALINE =    6/8528 [a.................................................]   0.07%
lbaselib.c
REF TIME A = 0:00    0 TIME B = 0:00    0 SPEED = -nan% BCOUNT =    8/8528 ALINE =    7/8528 [b.................................................]   0.08%
lbitlib.c
REF TIME A = 0:00    0 TIME B = 0:00    0 SPEED = -nan% BCOUNT =    9/8528 ALINE =    8/8528 [b.................................................]   0.09%
Converting src/mame/drivers/ymmu100.ppm...
REF TIME A = 0:00    0 TIME B = 0:00    0 SPEED = -nan% BCOUNT =   10/8528 ALINE =    9/8528 [b.................................................]   0.11%
lcode.c

...

Compiling src/mame/mame.cpp...
REF TIME A = 69:59 4199 TIME B = 41:05 2465 SPEED = 170.3% BCOUNT = 8524/8528 ALINE = 8525/8528 [#################################################Z]  99.96%
Compiling generated/version.cpp...
REF TIME A = 69:59 4199 TIME B = 41:05 2465 SPEED = 170.3% BCOUNT = 8525/8528 ALINE = 8524/8528 [#################################################Y]  99.95%
Building driver list...
REF TIME A = 69:59 4199 TIME B = 41:05 2465 SPEED = 170.3% BCOUNT = 8526/8528 ALINE = 8526/8528 [#################################################Z]  99.98%
Compiling generated/mame/mame/drivlist.cpp...
REF TIME A = 70:00 4200 TIME B = 41:05 2465 SPEED = 170.4% BCOUNT = 8527/8528 ALINE = 8527/8528 [#################################################Z]  99.99%
Linking mame64...
REF TIME A = 70:04 4204 TIME B = 41:09 2469 SPEED = 170.3% BCOUNT = 8528/8528 ALINE = 8528/8528 [##################################################] 100.00%
NOMATCH IN A
NOMATCH A:Creating ../../../../linux_gcc/bin/x64/Release/mame_mame<DUPLICATE LINE NUM=3>
NOMATCH IN B
DUPLICATES IN A
2x:"Generating m6509 disassembler source file..."
2x:""
2x:"Generating deco16 disassembler source file..."
2x:"Generating m65ce02 disassembler source file..."
2x:"Generating n2a03 disassembler source file..."
2x:"Generating m6510 disassembler source file..."
2x:"Generating r65c02 disassembler source file..."
2x:"Generating m4510 disassembler source file..."
2x:"Generating m65c02 disassembler source file..."
2x:"Generating m6502 disassembler source file..."
3x:"Creating ../../../../linux_gcc/bin/x64/Release/mame_mame"
2x:"Generating m740 disassembler source file..."
DUPLICATES IN B
2x:"Generating m6509 disassembler source file..."
2x:""
2x:"Generating deco16 disassembler source file..."
2x:"Generating m65ce02 disassembler source file..."
2x:"Generating n2a03 disassembler source file..."
2x:"Generating m6510 disassembler source file..."
2x:"Generating m740 disassembler source file..."
2x:"Generating r65c02 disassembler source file..."
2x:"Generating m4510 disassembler source file..."
2x:"Generating m6502 disassembler source file..."
2x:"Creating ../../../../linux_gcc/bin/x64/Release/mame_mame"
2x:"Generating m65c02 disassembler source file..."


I couldn't figure out why the line counts didn't match sometimes I wanted to "diff" the runs to see how the were different.

For instance, the first run A has 3 lines of "Creating ../../../../linux_gcc/bin/x64/Release/mame_mame" where the second run B has only 2 line of "Creating ../../../../linux_gcc/bin/x64/Release/mame_mame".

Re: Compile stats [Re: Golden Child] #112689
02/19/18 12:29 PM
02/19/18 12:29 PM
Joined: Feb 2014
Posts: 146
G
Golden Child Offline OP
Senior Member
Golden Child  Offline OP
Senior Member
G
Joined: Feb 2014
Posts: 146
And more silly compiling fun (under Ubuntu 17.10)

I thought I'd add some speech so that it'd say "Compile started" and "Compile Completed". Ubuntu's got this spd-say command which makes it easy, just do an os.execute("spd-say 'Open the pod bay doors, hal'")

And I wanted to see some graphics so why not try to send some values to gnuplot using lua, like this perl script from https://www.thanassis.space/gnuplotStreaming.html

and you get something like this:

[Linked Image]

and if you make a reference timefile you can just cat it to this script (which is fun to watch because it's fast):

cat luatimemake.txt | lua5.3 luaplot_test.lua

If you haven't got gnuplot installed, just do "sudo apt install gnuplot-qt".

Code

numbarks = 1

brun_table = {}
brun_linenumkey_table = {}


function saystring(str) str = "\""..str.."\"" print(str) os.execute("spd-say -w "..str)  end


function dogbark() for i = 1,numbarks do os.execute("play -q /usr/share/sounds/gnome/default/alerts/bark.ogg") end end
-- for dogbark to work you have to "sudo apt install sox"
dogbark()
saystring("Compile started")

f = io.open("luatimemake.txt","rb")
fileout = io.open("luacompareoutput2.txt","w")
linenum = 0

function formattime(a) local min,minfrac = math.modf(a/60) return min..":"..string.format("%02.0f",minfrac*60) end

function progressbar(size,fraction)
local numbars = math.floor((size-2)*(fraction))
local fracpart = ((size-2)*fraction)-numbars
--local onechar = string.sub("01234567890",math.floor(fracpart*10)+1,math.floor(fracpart*10)+1)
--local onechar = string.sub(".,;:-=+\\|/*#",math.floor(fracpart*10)+1,math.floor(fracpart*10)+1)
  local onechar = string.sub(".abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ",math.floor(fracpart*53)+1,math.floor(fracpart*53)+1)
local numremain = (size-2)-numbars
if numbars >= size - 2 then onechar = "" end
  return "["..string.rep("#",numbars)..onechar..string.rep(".",numremain-string.len(onechar)).."]".." "..string.format("%6.2f%%",100*fraction)
end

-- for i =0,100,0.5 do print (i,progressbar(80,i/100),progressbar(10,i/100)) end

function pprint(a) fileout:write(a .. "\n") print(a) end
function printt(t) for a,b in pairs(t) do print (a,b) end end

function dumpstring(a) for k = 1,string.len(a) do print (k,string.format("%3d",string.byte(a,k)).." "..string.sub(a,k,k)) end end

function add_to_table_and_check_duplicate(linestr,linetime,run_table,linenum,linenum_key_table)
local linestr2
if linestr ~= nil then
-- print(linestr) 
  if run_table[linestr]~= nil then 
    run_table[linestr]["count"] = run_table[linestr]["count"]+1  
    linestr2 = linestr .. "<DUPLICATE LINE NUM=" .. run_table[linestr]["count"]..">" print(linestr) 
  end
  if run_table[linestr]==nil then run_table[linestr] ={linenum=linenum,linetime=tonumber(linetime),count=1 }
                     else run_table[linestr2]={linenum=linenum,linetime=tonumber(linetime),count=1 }
                        pprint ("Original at "..run_table[linestr]["linenum"].." Duplicate at "..linenum..":\""..linestr2.."\" origtime = "..run_table[linestr]["linetime"].." newtime = "..linetime) 
  end 
end 
  if linestr2 ~= nil then 
     linenum_key_table[linenum]=linestr2
     run_table[linestr2]["linestr"]=linestr2
     return linestr2 
  else 
     linenum_key_table[linenum]=linestr
     run_table[linestr]["linestr"]=linestr
     return linestr 
  end
end



arun_linenumkey_table = {}
arun_table = {} f:seek("set") g = f:read() 
while g ~= nil do 
linetime,linestr = g:match("(.-)\t(.*)") 

maxval = tonumber(linetime) 
linenum = linenum + 1 

add_to_table_and_check_duplicate(linestr,linetime,arun_table,linenum,arun_linenumkey_table)

g = f:read() end

function setup_plot() 
pip = io.popen("gnuplot ","w")
end

function setup_plot2() 
pip2 = io.popen("gnuplot ","w")
end

function iif(a,b,c) if a then return b else return c end end

function make_plot2()

pip2:write("set xlabel 'time (in seconds)'\n")
pip2:write("set ylabel 'compile items'\n")
pip2:write("set style line 2 linewidth 2\n")

pip2:write("y2(x)=m2*x+b2\n")
pip2:write("set fit quiet\n")

pip2:write("fit y2(x) '-' via m2,b2\n")

for i =1,#brun_linenumkey_table,iif(#brun_linenumkey_table<skipinterval,1,skipinterval*skipfitmultiplier) do 
pip2:write(brun_table[brun_linenumkey_table[i]]["linetime"].." "..brun_table[brun_linenumkey_table[i]]["linenum"].."\n")
end
pip2:write("e\n")
pip2:flush()

pip2:write("plot '-' title 'brun' with lines,y2(x)\n")

for i =#brun_linenumkey_table-50,#brun_linenumkey_table do 
if i>=1 then
pip2:write(brun_table[brun_linenumkey_table[i]]["linetime"].." "..brun_table[brun_linenumkey_table[i]]["linenum"].."\n")
end
end
pip2:write("e\n")
pip2:flush()

end

skipinterval = 10 -- lets skip to plot less datapoints, make it faster
skipmultiplier = 2
skipfitmultiplier = 5
function make_plot()

pip:write("set xlabel 'time (in seconds)'\n")
pip:write("set ylabel 'compile items'\n")

pip:write("y(x)=m*x+b\n")
pip:write("set fit quiet\n")
pip:write("fit y(x) '-' via m,b\n")

for i =1,#arun_linenumkey_table,skipinterval*skipfitmultiplier do 
pip:write(arun_table[arun_linenumkey_table[i]]["linetime"].." "..arun_table[arun_linenumkey_table[i]]["linenum"].."\n")
end
pip:write("e\n")
pip:flush()


pip:write("y2(x)=m2*x+b2\n")
pip:write("fit y2(x) '-' via m2,b2\n")


for i =1,#brun_linenumkey_table,skipinterval*skipfitmultiplier do 
pip:write(brun_table[brun_linenumkey_table[i]]["linetime"].." "..brun_table[brun_linenumkey_table[i]]["linenum"].."\n")
end
pip:write("e\n")
pip:flush()


pip:write("set style line 2 linewidth 2\n")
pip:write("set style line 3 linewidth 6\n")
pip:write("set yrange [*:10000]\n")
--pip:write("plot y(x), '-' with lines, '-' with points\n")
pip:write("plot y(x), '-' title 'arun' with lines, '-' title 'brun' with lines ls 3, y2(x)\n")


for i =1,#arun_linenumkey_table,skipinterval do 
pip:write(arun_table[arun_linenumkey_table[i]]["linetime"].." "..arun_table[arun_linenumkey_table[i]]["linenum"].."\n")
end
pip:write("e\n")
pip:flush()

for i =1,#brun_linenumkey_table,skipinterval do 
--print (i,brun_linenumkey_table[i],brun_table[brun_linenumkey_table[i]]["linetime"],brun_table[brun_linenumkey_table[i]]["linenum"])
pip:write(brun_table[brun_linenumkey_table[i]]["linetime"].." "..brun_table[brun_linenumkey_table[i]]["linenum"].."\n")
end
pip:write("e\n")
pip:flush()

end -- makeplot

setup_plot()
make_plot()

setup_plot2()

alinecount = 0
for a,b in pairs(arun_table) do alinecount = alinecount + 1 end
print("TOTAL ITEMS:" .. alinecount)

blinecount = 0

brun_table = {}
brun_linenumkey_table = {}

starttime = os.time()


while true do

if (blinecount % (skipinterval*skipmultiplier)) == 0 then make_plot() end

if (blinecount % 2) == 0 then make_plot2() end

newline = io.read()
blinecount = blinecount + 1



if newline == nil then break end

if string.match(newline,"(Linking mame64)") then saystring(string.match(newline,"(Linking mame64)")) end

if newline:match("\t") ~= nil then
  print("matching TAB character")
  newlinetime,newline = newline:match("(.-)\t(.*)") 
  if newlinetime=="" then print("newlinetime is empty string") newlinetime = os.time() - starttime print(newlinetime) end
  if newlinetime==nil then print("newlinetime is nil") newlinetime = os.time() - starttime print(newlinetime) end
else
  newlinetime = os.time() - starttime
end

newline = add_to_table_and_check_duplicate(newline,newlinetime,brun_table,blinecount,brun_linenumkey_table)

if arun_table[newline] ~= nil then oldlinetime = arun_table[newline]["linetime"] oldlinenum = arun_table[newline]["linenum"] else oldlinetime = nil end

pprint (newline)

if oldlinetime == nil then pprint("*** line not found *** NEW >>> "..newline) print(arun_table[newline]) else pprint("REF TIME A = " .. formattime(oldlinetime).." "..string.format("%4d",oldlinetime) .. " TIME B = " .. formattime(newlinetime).." ".. string.format("%4d",newlinetime) .. " SPEED = " .. string.format("%2.1f",(oldlinetime / newlinetime)*100) .."%".. " BCOUNT = " .. string.format("%4d",blinecount) .."/"..string.format("%4d",alinecount).." ".."ALINE = " .. string.format("%4d",oldlinenum) .."/"..string.format("%4d",alinecount).." ".. progressbar(52,oldlinenum/alinecount)) 
   brun_table[newline]["match"]=1
   arun_table[newline]["match"]=1
end
end

print("NOMATCH IN A")
for c,d in pairs(arun_table) do if arun_table[c]["match"]==nil then print ("NOMATCH A:"..c) end end
print("NOMATCH IN B")
for c,d in pairs(brun_table) do if brun_table[c]["match"]==nil then print ("NOMATCH B:"..c) end end

print("DUPLICATES IN A")
for c,d in pairs(arun_table) do if arun_table[c]["count"]>1 then print (arun_table[c]["count"].."x:".."line "..arun_table[c]["linenum"]..":".."\""..c.."\"") end end

print("DUPLICATES IN A IN ORDER")
for i =1,#arun_linenumkey_table do if arun_table[arun_linenumkey_table[i]]["count"]>1 or string.find(arun_table[arun_linenumkey_table[i]]["linestr"],"<DUPLICATE LINE NUM=") ~= nil then print("A DUP:"..arun_table[arun_linenumkey_table[i]]["count"].."x:".."line "..arun_table[arun_linenumkey_table[i]]["linenum"]..":".."\""..arun_linenumkey_table[i].."\"") end end

print("DUPLICATES IN B")
for c,d in pairs(brun_table) do if brun_table[c]["count"]>1 then print (brun_table[c]["count"].."x:".."line "..arun_table[c]["linenum"]..":".."\""..c.."\"") end end

print("DUPLICATES IN B IN ORDER")
for i =1,#brun_linenumkey_table do if brun_table[brun_linenumkey_table[i]]["count"]>1 or string.find(brun_table[brun_linenumkey_table[i]]["linestr"],"<DUPLICATE LINE NUM=") ~= nil then print("B DUP:"..brun_table[arun_linenumkey_table[i]]["count"].."x:".."line "..brun_table[brun_linenumkey_table[i]]["linenum"]..":".."\""..brun_linenumkey_table[i].."\"") end end

print("NUM ENTRIES IN A: "..#arun_linenumkey_table)
print("NUM ENTRIES IN B: "..#brun_linenumkey_table)

compilemins,compileminsfrac = math.modf((os.time()-starttime)/60)
compilesecs = compileminsfrac * 60
saystring("Compile completed in "..string.format("%.0f",compilemins).." minutes and "..string.format("%.0f",compilesecs).." seconds.")

dogbark()

os.execute("sleep 10")

pip:close()
pip2:close()


Re: Compile stats [Re: Golden Child] #112767
02/28/18 03:24 PM
02/28/18 03:24 PM
Joined: Feb 2014
Posts: 146
G
Golden Child Offline OP
Senior Member
Golden Child  Offline OP
Senior Member
G
Joined: Feb 2014
Posts: 146
So I wanted to add some labels to the graphs, but they overlapped so you couldn't read them. Why not rotate the text by a few degrees with gnuplot? It makes a cool radial pattern. More silly fun.

The graphs seem to slow the compile by about 2-3%, it added 1:30 to the original compile time of 70:04.

REF TIME A = 70:04 4204 TIME B = 71:32 4292 SPEED = 97.9% BCOUNT = 8527/8528 ALINE = 8528/8528 [##################################################] 100.00%

[Linked Image]

Code
-- luacompilegraph.lua
--
-- "It's time for your graph, Master Joey"

numbarks = 1

brun_table = {}
brun_linenumkey_table = {}


function saystring(str) str = "\""..str.."\"" print(str) os.execute("spd-say -w "..str)  end


function dogbark() for i = 1,numbarks do os.execute("play -q /usr/share/sounds/gnome/default/alerts/bark.ogg") end end
-- for dogbark to work you have to "sudo apt install sox"
dogbark()
saystring("Compile started")

f = io.open("luatimemake.txt","rb")
fileout = io.open("luacompareoutput2.txt","w")
linenum = 0

function formattime(a) local min,minfrac = math.modf(a/60) return min..":"..string.format("%02.0f",minfrac*60) end

function progressbar(size,fraction)
local numbars = math.floor((size-2)*(fraction))
local fracpart = ((size-2)*fraction)-numbars
--local onechar = string.sub("01234567890",math.floor(fracpart*10)+1,math.floor(fracpart*10)+1)
--local onechar = string.sub(".,;:-=+\\|/*#",math.floor(fracpart*10)+1,math.floor(fracpart*10)+1)
  local onechar = string.sub(".abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ",math.floor(fracpart*53)+1,math.floor(fracpart*53)+1)
local numremain = (size-2)-numbars
if numbars >= size - 2 then onechar = "" end
  return "["..string.rep("#",numbars)..onechar..string.rep(".",numremain-string.len(onechar)).."]".." "..string.format("%6.2f%%",100*fraction) --..fracpart..numbars
end

-- for i =0,100,0.5 do print (i,progressbar(80,i/100),progressbar(10,i/100)) end

function pprint(a) fileout:write(a .. "\n") print(a) end
function printt(t) for a,b in pairs(t) do print (a,b) end end

function dumpstring(a) for k = 1,string.len(a) do print (k,string.format("%3d",string.byte(a,k)).." "..string.sub(a,k,k)) end end


--[[

run_table

entries keyed on the text of the line

each entry consists of a table where:

["linestr"] = text of line
["linenum"] = line number
["linetime"] = line time
["count"] = line count of how many times line appears (>1 if duplicates)
["match"] = matched in another table


so for example, accessing the various properties of a given line would be:
run_table["text key of a line"]["linenum"] = 32
run_table["text key of a line"]["linetime"] = 3600
run_table["text key of a line"]["count"] = 5

--]]



function add_to_table_and_check_duplicate(linestr,linetime,run_table,linenum,linenum_key_table)
local linestr2
if linestr ~= nil then
-- print(linestr) 
  if run_table[linestr]~= nil then 
    run_table[linestr]["count"] = run_table[linestr]["count"]+1  
    linestr2 = linestr .. "<DUPLICATE LINE NUM=" .. run_table[linestr]["count"]..">" print(linestr) 
  end
  if run_table[linestr]==nil then run_table[linestr] ={linenum=linenum,linetime=tonumber(linetime),count=1 }
                     else run_table[linestr2]={linenum=linenum,linetime=tonumber(linetime),count=1 }
                        pprint ("Original at "..run_table[linestr]["linenum"].." Duplicate at "..linenum..":\""..linestr2.."\" origtime = "..run_table[linestr]["linetime"].." newtime = "..linetime) 
  end 
end 
  if linestr2 ~= nil then 
     linenum_key_table[linenum]=linestr2
     run_table[linestr2]["linestr"]=linestr2
     return linestr2 
  else 
     linenum_key_table[linenum]=linestr
     run_table[linestr]["linestr"]=linestr
     return linestr 
  end
end



arun_linenumkey_table = {}
arun_table = {} f:seek("set") g = f:read() 
while g ~= nil do 
linetime,linestr = g:match("(.-)\t(.*)") 

maxval = tonumber(linetime) 
linenum = linenum + 1 

add_to_table_and_check_duplicate(linestr,linetime,arun_table,linenum,arun_linenumkey_table)

g = f:read() end


--luaplot_test

function escape_quotes(s) return string.gsub(s,"\"","\\\"") end


function setup_plot() 
-- Open a pipe to gnuplot for writing
pip = io.popen("gnuplot ","w")
end

function setup_plot2() 
-- Open another pipe to gnuplot for writing
pip2 = io.popen("gnuplot ","w")
end

function iif(a,trueresult,falseresult) if a then return trueresult else return falseresult end end

function make_plot2()
pip2:write("set xlabel 'time (in seconds)'\n")
pip2:write("set ylabel 'compile items'\n")
pip2:write("set style line 2 linewidth 2\n")

bottomedge = #brun_linenumkey_table - 50
--pip2:write("set yrange ["..bottomedge-(bottomedge % 50) ..":*]\n")
--io.write("set xrange ["..(#brun_linenumkey_table % 50) ..":*]\n")

pip2:write("y2(x)=m2*x+b2\n")
pip2:write("set fit quiet\n")

pip2:write("fit y2(x) '-' via m2,b2\n")

for i =1,#brun_linenumkey_table,iif(#brun_linenumkey_table<skipinterval,1,skipinterval*skipfitmultiplier) do 
pip2:write(brun_table[brun_linenumkey_table[i]]["linetime"].." "..brun_table[brun_linenumkey_table[i]]["linenum"].."\n")
end
pip2:write("e\n")
pip2:flush()

-- set offset left, right, top, bottom  
-- we want some white space to the right so let's try a 0.7 margin and 0.1 margin for everything else
pip2:write("set offset graph 0.1, graph 0.7, graph 0.1, graph 0.1\n")

num_labels=10

-- unset label clears all of the labels previously set
pip2:write("unset label\n")
-- now lets get the last few items and create some labels for them, rotating them so we can see them better
for i =#brun_linenumkey_table-num_labels,#brun_linenumkey_table do 
if i>=1 then
pip2:write("set label "..i.." \""..escape_quotes(brun_table[brun_linenumkey_table[i]]["linestr"]).."\" at "..brun_table[brun_linenumkey_table[i]]["linetime"]..","..brun_table[brun_linenumkey_table[i]]["linenum"].." rotate by 90-"..#brun_linenumkey_table-i.."*25\n")
end
end
pip2:flush()

pip2:write("plot '-' title 'brun' with linespoints,y2(x)\n")

for i =#brun_linenumkey_table-50,#brun_linenumkey_table do 
if i>=1 then
pip2:write(brun_table[brun_linenumkey_table[i]]["linetime"].." "..brun_table[brun_linenumkey_table[i]]["linenum"].."\n")
end
end
pip2:write("e\n")
pip2:flush()
end -- make_plot2


skipinterval = 10 -- lets skip to plot less datapoints, make it faster
skipmultiplier = 2
skipfitmultiplier = 5

function make_plot()

pip:write("set xlabel 'time (in seconds)'\n")
pip:write("set ylabel 'compile items'\n")

pip:write("y(x)=m*x+b\n")
pip:write("set fit quiet\n")
pip:write("fit y(x) '-' via m,b\n")

for i =1,#arun_linenumkey_table,skipinterval*skipfitmultiplier do 
pip:write(arun_table[arun_linenumkey_table[i]]["linetime"].." "..arun_table[arun_linenumkey_table[i]]["linenum"].."\n")
end
pip:write("e\n")
pip:flush()


pip:write("y2(x)=m2*x+b2\n")
pip:write("fit y2(x) '-' via m2,b2\n")


for i =1,#brun_linenumkey_table,skipinterval*skipfitmultiplier do 
--print (i,brun_linenumkey_table[i],brun_table[brun_linenumkey_table[i]]["linetime"],brun_table[brun_linenumkey_table[i]]["linenum"])
pip:write(brun_table[brun_linenumkey_table[i]]["linetime"].." "..brun_table[brun_linenumkey_table[i]]["linenum"].."\n")
end
pip:write("e\n")
pip:flush()


pip:write("set style line 2 linewidth 2\n")
pip:write("set style line 3 linewidth 6\n")
pip:write("set yrange [*:10000]\n")
--pip:write("plot y(x), '-' with lines, '-' with points\n")
pip:write("plot y(x), '-' title 'arun' with lines, '-' title 'brun' with lines ls 3, y2(x)\n")


for i =1,#arun_linenumkey_table,skipinterval do 
pip:write(arun_table[arun_linenumkey_table[i]]["linetime"].." "..arun_table[arun_linenumkey_table[i]]["linenum"].."\n")
end
pip:write("e\n")
pip:flush()

for i =1,#brun_linenumkey_table,skipinterval do 
--print (i,brun_linenumkey_table[i],brun_table[brun_linenumkey_table[i]]["linetime"],brun_table[brun_linenumkey_table[i]]["linenum"])
pip:write(brun_table[brun_linenumkey_table[i]]["linetime"].." "..brun_table[brun_linenumkey_table[i]]["linenum"].."\n")
end
pip:write("e\n")
-- don't forget to flush
pip:flush()

end -- makeplot

setup_plot()
make_plot()

setup_plot2()

-- os.execute("sleep 10")


alinecount = 0
for a,b in pairs(arun_table) do alinecount = alinecount + 1 end
print("TOTAL ITEMS:" .. alinecount)

blinecount = 0

brun_table = {}
brun_linenumkey_table = {}

starttime = os.time()


while true do


-- do our makeplot every <skipinterval> lines
if (blinecount % (skipinterval*skipmultiplier)) == 0 then make_plot() end

-- lets do a make_plot2 every 2 lines
if (blinecount % 2) == 0 then make_plot2() end

newline = io.read()
blinecount = blinecount + 1



if newline == nil then break end

if string.match(newline,"(Linking mame64)") then saystring(string.match(newline,"(Linking mame64)")) end

if newline:match("\t") ~= nil then
  print("matching TAB character")
  newlinetime,newline = newline:match("(.-)\t(.*)") 
  if newlinetime=="" then print("newlinetime is empty string") newlinetime = os.time() - starttime print(newlinetime) end
  if newlinetime==nil then print("newlinetime is nil") newlinetime = os.time() - starttime print(newlinetime) end
else
  newlinetime = os.time() - starttime
end

newline = add_to_table_and_check_duplicate(newline,newlinetime,brun_table,blinecount,brun_linenumkey_table)

if arun_table[newline] ~= nil then oldlinetime = arun_table[newline]["linetime"] oldlinenum = arun_table[newline]["linenum"] else oldlinetime = nil end

pprint (newline)

if oldlinetime == nil then pprint("*** line not found *** NEW >>> "..newline) print(arun_table[newline]) else pprint("REF TIME A = " .. formattime(oldlinetime).." "..string.format("%4d",oldlinetime) .. " TIME B = " .. formattime(newlinetime).." ".. string.format("%4d",newlinetime) .. " SPEED = " .. string.format("%2.1f",(oldlinetime / newlinetime)*100) .."%".. " BCOUNT = " .. string.format("%4d",blinecount) .."/"..string.format("%4d",alinecount).." ".."ALINE = " .. string.format("%4d",oldlinenum) .."/"..string.format("%4d",alinecount).." ".. progressbar(52,oldlinenum/alinecount)) 
   brun_table[newline]["match"]=1
   arun_table[newline]["match"]=1
end
end

print("NOMATCH IN A")
for c,d in pairs(arun_table) do if arun_table[c]["match"]==nil then print ("NOMATCH A:"..c) end end
print("NOMATCH IN B")
for c,d in pairs(brun_table) do if brun_table[c]["match"]==nil then print ("NOMATCH B:"..c) end end

print("DUPLICATES IN A")
for c,d in pairs(arun_table) do if arun_table[c]["count"]>1 then print (arun_table[c]["count"].."x:".."line "..arun_table[c]["linenum"]..":".."\""..c.."\"") end end

print("DUPLICATES IN A IN ORDER")
for i =1,#arun_linenumkey_table do if arun_table[arun_linenumkey_table[i]]["count"]>1 or string.find(arun_table[arun_linenumkey_table[i]]["linestr"],"<DUPLICATE LINE NUM=") ~= nil then print("A DUP:"..arun_table[arun_linenumkey_table[i]]["count"].."x:".."line "..arun_table[arun_linenumkey_table[i]]["linenum"]..":".."\""..arun_linenumkey_table[i].."\"") end end

print("DUPLICATES IN B")
for c,d in pairs(brun_table) do if brun_table[c]["count"]>1 then print (brun_table[c]["count"].."x:".."line "..arun_table[c]["linenum"]..":".."\""..c.."\"") end end

print("DUPLICATES IN B IN ORDER")
for i =1,#brun_linenumkey_table do if brun_table[brun_linenumkey_table[i]]["count"]>1 or string.find(brun_table[brun_linenumkey_table[i]]["linestr"],"<DUPLICATE LINE NUM=") ~= nil then print("B DUP:"..brun_table[arun_linenumkey_table[i]]["count"].."x:".."line "..brun_table[brun_linenumkey_table[i]]["linenum"]..":".."\""..brun_linenumkey_table[i].."\"") end end

print("NUM ENTRIES IN A: "..#arun_linenumkey_table)
print("NUM ENTRIES IN B: "..#brun_linenumkey_table)


compilemins,compileminsfrac = math.modf((os.time()-starttime)/60)
compilesecs = compileminsfrac * 60
saystring("Compile completed in "..string.format("%.0f",compilemins).." minutes and "..string.format("%.0f",compilesecs).." seconds.")

dogbark()

-- Pause so we can look at the graphs for 10 seconds because they disappear when gnuplot closes
os.execute("sleep 10")

pip:close()
pip2:close()


Re: Compile stats [Re: Golden Child] #112889
03/16/18 12:21 PM
03/16/18 12:21 PM
Joined: Feb 2014
Posts: 146
G
Golden Child Offline OP
Senior Member
Golden Child  Offline OP
Senior Member
G
Joined: Feb 2014
Posts: 146
So how much memory do you really need to compile mame efficiently?

/usr/bin/time make -j2 tells you things like:

10473.27user 1103.76system 1:43:29elapsed 186%CPU (0avgtext+0avgdata 2165100maxresident)k
3767160inputs+5861408outputs (10198major+253164785minor)pagefaults 0swaps

And it looks like you can compile with 2GB memory since it says 2165100maxresident, but it seems to need at least 3GB memory to do it well.

I thought I'd run a little experiment with different amounts of memory since I had a Dell Optiplex 745 Core2Duo system and some spare DIMMS, but you can simulate having less memory with a mem=2G or mem=3G kernel parameter at boot time (it's a lot easier than pulling dimms out).

I installed Ubuntu to an external USB drive and directed the Dell Optiplex "F12 boot menu" to boot up with it. It's a neat way to try the same Ubuntu system on a bunch of different hardware, you just move the external USB from system to system and run the same exact setup.

My lua script that gives timing into will emit a "DONE - COMPILE COMPLETE" line when it receives nil along with some other info which is useful when comparing systems:

Code

-- luatime2.lua
-- This will take the input and print the time in seconds before the line.
-- Once complete it will print info about your system and run mame for a few seconds.

donestr = "DONE - COMPILE COMPLETE"

function iif(b,t,f) if b then return t else return f end end 
start = os.time() 

while true do 
  linestr = io.read() 
  linetime = os.time() 
  print (linetime-start,iif(linestr==nil,donestr,linestr)) 
  if linestr==nil then break end 
end 

function print_and_execute(str) print(string.rep("=",80).."\n".."EXECUTING: "..str) os.execute(str) end

print_and_execute("lscpu | grep \"Model name:\"")
print_and_execute("lscpu")
print_and_execute("lspci -nn | grep -E -i \"Display|VGA\"")
print_and_execute("lspci -nn")
print_and_execute("ls -l ./mame64")
print_and_execute("md5sum ./mame64")
print_and_execute("gcc --version | head -n 1")
print_and_execute("lsb_release -a")
print_and_execute("uname -a")
print_and_execute("date")
print_and_execute("lsmod | grep \"video\\|drm\\|radeon\\|nouveau\"")
print_and_execute("cat /proc/meminfo | grep \"MemTotal\"")

mamebasestr = "/usr/bin/time ./mame64 -seconds_to_run 7 -sdlvideofps"

mamegamestr = "pacman"

print_and_execute(mamebasestr.." "..mamegamestr.." -video opengl -verbose")



And you can invoke it with:

Code
make clean; /usr/bin/time make -j2 2>&1 | lua5.3 lua_time2.lua | tee  optiplex745_core2_6700_7gb_4mbl2_j2compile.txt | nl

So the 2>&1 part just redirects standard error to stdout, so we get the output of /usr/bin/time
tee just writes the output to a file
nl just adds a line number to the beginning of the line

And what did I discover:

Code
grep "DONE\|%CPU\|swap\|Model name\|MemTotal:"  optiplex*.txt


On a 2GB system: We run make -j2 to use both cores and it get 77% CPU and takes 4:17)

optiplex745_core2_6700_2gb_4mbl2_compile.txt:15438	10721.09user 1264.19system 4:17:17elapsed 77%CPU (0avgtext+0avgdata 1574196maxresident)k
optiplex745_core2_6700_2gb_4mbl2_compile.txt:15438	41428312inputs+5904776outputs (874171major+253765634minor)pagefaults 0swaps
optiplex745_core2_6700_2gb_4mbl2_compile.txt:15438	DONE - COMPILE COMPLETE
optiplex745_core2_6700_2gb_4mbl2_compile.txt:EXECUTING: lscpu | grep "Model name:"
optiplex745_core2_6700_2gb_4mbl2_compile.txt:Model name:            Intel(R) Core(TM)2 CPU          6700  @ 2.66GHz
optiplex745_core2_6700_2gb_4mbl2_compile.txt:Model name:            Intel(R) Core(TM)2 CPU          6700  @ 2.66GHz
optiplex745_core2_6700_2gb_4mbl2_compile.txt:MemTotal:        2037108 kB

But here's a surprise, make -j1 runs FASTER with 82% cpu in 3:34 on a 2GB system.

optiplex745_core2_6700_2gb_4mbl2_mem2g_j1compile.txt:12846	9535.24user 1019.71system 3:34:06elapsed 82%CPU (0avgtext+0avgdata 1610620maxresident)k
optiplex745_core2_6700_2gb_4mbl2_mem2g_j1compile.txt:12846	23379376inputs+5905032outputs (267625major+253551458minor)pagefaults 0swaps
optiplex745_core2_6700_2gb_4mbl2_mem2g_j1compile.txt:12846	DONE - COMPILE COMPLETE
optiplex745_core2_6700_2gb_4mbl2_mem2g_j1compile.txt:EXECUTING: lscpu | grep "Model name:"
optiplex745_core2_6700_2gb_4mbl2_mem2g_j1compile.txt:Model name:            Intel(R) Core(TM)2 CPU          6700  @ 2.66GHz
optiplex745_core2_6700_2gb_4mbl2_mem2g_j1compile.txt:Model name:            Intel(R) Core(TM)2 CPU          6700  @ 2.66GHz
optiplex745_core2_6700_2gb_4mbl2_mem2g_j1compile.txt:MemTotal:        2047352 kB

A 3GB system, make -j2 will complete in 1:43 with 186% cpu .  Major faults require disk access, and we only get 10198 major here.

optiplex745_core2_6700_3gb_4mbl2_compile.txt:6210	10473.27user 1103.76system 1:43:29elapsed 186%CPU (0avgtext+0avgdata 2165100maxresident)k
optiplex745_core2_6700_3gb_4mbl2_compile.txt:6210	3767160inputs+5861408outputs (10198major+253164785minor)pagefaults 0swaps
optiplex745_core2_6700_3gb_4mbl2_compile.txt:6210	DONE - COMPILE COMPLETE
optiplex745_core2_6700_3gb_4mbl2_compile.txt:EXECUTING: lscpu | grep "Model name:"
optiplex745_core2_6700_3gb_4mbl2_compile.txt:Model name:            Intel(R) Core(TM)2 CPU          6700  @ 2.66GHz
optiplex745_core2_6700_3gb_4mbl2_compile.txt:Model name:            Intel(R) Core(TM)2 CPU          6700  @ 2.66GHz
optiplex745_core2_6700_3gb_4mbl2_compile.txt:MemTotal:        3069300 kB

A 7GB system, make -j2 will complete in 1:39 with 192% cpu .  Only 348 major faults and 192% CPU usage.

optiplex745_core2_6700_7gb_4mbl2_j2compile.txt:5971	10467.70user 1053.96system 1:39:30elapsed 192%CPU (0avgtext+0avgdata 2239380maxresident)k
optiplex745_core2_6700_7gb_4mbl2_j2compile.txt:5971	925896inputs+5781240outputs (348major+253878672minor)pagefaults 0swaps
optiplex745_core2_6700_7gb_4mbl2_j2compile.txt:5971	DONE - COMPILE COMPLETE
optiplex745_core2_6700_7gb_4mbl2_j2compile.txt:EXECUTING: lscpu | grep "Model name:"
optiplex745_core2_6700_7gb_4mbl2_j2compile.txt:Model name:            Intel(R) Core(TM)2 CPU          6700  @ 2.66GHz
optiplex745_core2_6700_7gb_4mbl2_j2compile.txt:Model name:            Intel(R) Core(TM)2 CPU          6700  @ 2.66GHz
optiplex745_core2_6700_7gb_4mbl2_j2compile.txt:MemTotal:        7066468 kB



So I'd probably say that 3GB is probably the minimum amount of memory to do it without a bunch of thrashing.

This is compiling mame 194 on Ubuntu 16.04 LTS 64bit on a Core2Duo 6700.

Re: Compile stats [Re: Golden Child] #112939
03/22/18 03:49 AM
03/22/18 03:49 AM
Joined: Feb 2014
Posts: 146
G
Golden Child Offline OP
Senior Member
Golden Child  Offline OP
Senior Member
G
Joined: Feb 2014
Posts: 146
So I thought I'd see what would happen if I compiled mame 194 under 32 bit i386 Debian 9.3. I was actually surprised how much trouble I had compiling it. I was able to do it perfectly fine with a 2GB machine under ubuntu 16.04 with GCC 5.4. My laptop was slow, but it worked fine.

Code
13517	Compiling src/mame/mame.cpp...
13517	Building driver list...
13519	34922 driver(s) found
13519	Compiling generated/version.cpp...
13519	Compiling generated/mame/mame/drivlist.cpp...
13529	Linking mame...
13809	24266.71user 1461.97system 3:50:09elapsed 186%CPU (0avgtext+0avgdata 1319532maxresident)k
13809	3621416inputs+4597848outputs (5762major+166197861minor)pagefaults 0swaps
13809	DONE - COMPILE COMPLETE
================================================================================
EXECUTING: lscpu | grep "Model name:"
Model name:            Genuine Intel(R) CPU           U2500  @ 1.20GHz
================================================================================
EXECUTING: lscpu
Architecture:          i686
CPU op-mode(s):        32-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    1
Core(s) per socket:    2
Socket(s):             1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 14
Model name:            Genuine Intel(R) CPU           U2500  @ 1.20GHz
Stepping:              8
CPU MHz:               800.000
CPU max MHz:           1200.0000
CPU min MHz:           800.0000
BogoMIPS:              2394.09
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              2048K
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx constant_tsc arch_perfmon bts aperfmperf pni monitor vmx est tm2 xtpr pdcm retpoline dtherm

================================================================================
EXECUTING: ls -l ./mame
-rwxrwxr-x 1 golden golden 226113840 Mar 18 14:54 ./mame
================================================================================
EXECUTING: md5sum ./mame
9f8a7cc185264d5da6df2fcc9a3d5d8d  ./mame
================================================================================
EXECUTING: gcc --version | head -n 1
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609
================================================================================
EXECUTING: lsb_release -a
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.4 LTS
Release:	16.04
Codename:	xenial
================================================================================
EXECUTING: uname -a
Linux  4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:22:43 UTC 2018 i686 i686 i686 GNU/Linux
================================================================================
EXECUTING: date
Sun Mar 18 10:29:30 PDT 2018
================================================================================




But I had lots of problems compiling under Debian 9.4, specifically the barcrest and the bfm subtargets seemed to cause virtual memory exhaustion. On this system with 4GB, under Debian i686 it only has 3GB and I have a 10GB swapfile. However it only seems to have a 3GB maximum address space.

I finally got it to compile by commenting out the barcrest and bfm subtargets in arcade.lua (scripts/target/mame/arcade.lua)


Code
function linkProjects_mame_arcade(_target, _subtarget)
	links {
		"acorn",
		"adp",
		"alba",
		"alliedl",
		"alpha",
		"amiga",
		"aristocr",
		"ascii",
		"atari",
		"atlus",
	--	"barcrest",       COMMENTED OUT  TWO DASHES
	--	"bfm",        COMMENTED OUT
		"bmc",
		"capcom",
		"ces",
	

--[[     COMMENTED OUT TWO DASHES AND TWO BRACKETS
createMAMEProjects(_target, _subtarget, "barcrest")
files {
	MAME_DIR .. "src/mame/drivers/mpu2.cpp",
	MAME_DIR .. "src/mame/drivers/mpu3.cpp",
	MAME_DIR .. "src/mame/machine/mpu4.cpp",
	MAME_DIR .. "src/mame/includes/mpu4.h",
	MAME_DIR .. "src/mame/drivers/mpu4.cpp",
	MAME_DIR .. "src/mame/drivers/mpu4avan.cpp",
	MAME_DIR .. "src/mame/drivers/mpu4bwb.cpp",
	MAME_DIR .. "src/mame/drivers/mpu4concept.cpp",
	MAME_DIR .. "src/mame/drivers/mpu4crystal.cpp",
	MAME_DIR .. "src/mame/drivers/mpu4dealem.cpp",
	MAME_DIR .. "src/mame/drivers/mpu4empire.cpp",
	MAME_DIR .. "src/mame/drivers/mpu4mdm.cpp",
	MAME_DIR .. "src/mame/drivers/mpu4misc.cpp",
	MAME_DIR .. "src/mame/drivers/mpu4mod2sw.cpp",
	MAME_DIR .. "src/mame/drivers/mpu4mod4yam.cpp",
	MAME_DIR .. "src/mame/drivers/mpu4plasma.cpp",
	MAME_DIR .. "src/mame/drivers/mpu4sw.cpp",
	MAME_DIR .. "src/mame/drivers/mpu4union.cpp",
	MAME_DIR .. "src/mame/drivers/mpu4vid.cpp",
	MAME_DIR .. "src/mame/drivers/mpu5.cpp",
	MAME_DIR .. "src/mame/drivers/mpu5.hxx",
}

--]]
--[[
createMAMEProjects(_target, _subtarget, "bfm")
files {
	MAME_DIR .. "src/mame/drivers/bfcobra.cpp",
	MAME_DIR .. "src/mame/machine/bfm_comn.cpp",
	MAME_DIR .. "src/mame/machine/bfm_comn.h",
	MAME_DIR .. "src/mame/drivers/bfm_sc1.cpp",
	MAME_DIR .. "src/mame/drivers/bfm_sc2.cpp",
	MAME_DIR .. "src/mame/video/bfm_adr2.cpp",
	MAME_DIR .. "src/mame/video/bfm_adr2.h",
	MAME_DIR .. "src/mame/drivers/bfm_sc4.cpp",
	MAME_DIR .. "src/mame/drivers/bfm_sc5.cpp",
	MAME_DIR .. "src/mame/includes/bfm_sc5.h",
	MAME_DIR .. "src/mame/drivers/bfm_sc5sw.hxx",
	MAME_DIR .. "src/mame/drivers/bfm_ad5.cpp",
	MAME_DIR .. "src/mame/includes/bfm_ad5.h",
	MAME_DIR .. "src/mame/drivers/bfm_ad5sw.hxx",
	MAME_DIR .. "src/mame/machine/bfm_sc45_helper.cpp",
	MAME_DIR .. "src/mame/machine/bfm_sc45_helper.h",
	MAME_DIR .. "src/mame/includes/bfm_sc4.h",
	MAME_DIR .. "src/mame/drivers/bfm_swp.cpp",
	MAME_DIR .. "src/mame/drivers/bfmsys83.cpp",
	MAME_DIR .. "src/mame/drivers/bfmsys85.cpp",
	MAME_DIR .. "src/mame/machine/sec.cpp",
	MAME_DIR .. "src/mame/machine/sec.h",
	MAME_DIR .. "src/mame/machine/bfm_bd1.cpp",
	MAME_DIR .. "src/mame/machine/bfm_bd1.h",
	MAME_DIR .. "src/mame/machine/bfm_bda.cpp",
	MAME_DIR .. "src/mame/machine/bfm_bda.h",
	MAME_DIR .. "src/mame/video/bfm_dm01.cpp",
	MAME_DIR .. "src/mame/video/bfm_dm01.h",
	MAME_DIR .. "src/mame/drivers/rastersp.cpp",
}
--]]


Then I had to remove the items from mame.lst as well (src/mame/mame.lst) so they wouldn't get added to drvlist.cpp

Code

/*  
@source:bfcobra.cpp
beeline                         // 1991 BFM
escounts                        // 1990 BFM
inquiztr                        // 1989 BFM
qos                             // 1992 BFM
qosa                            // 1992 BFM
qosb                            // 1992 BFM
quizvadr                        // 1991 BFM
trebltop                        // 1991 BFM

...

@source:bfmsys85.cpp
b85bdclb                        //
b85bdclba                       //
...                 
b85ritzd                        //
b85royal                        //
b85scard                        // (c) 198? BFM
b85sngam                        //
*/


/
@source:mpu2.cpp
m2hilite                        // Hi-Lights (Barcrest) (MPU2)
m2svlite                        // Silver Lights (Barcrest) (MPU2)

...

@source:mpu5.cpp
m55050                          // Fifty Fifty (Bwb)
m5aceclb                        // Ace Of Clubs (Empire)
m5aceclba                       //
m5aceclbb                       //
m5addams                        // Addams Family (Barcrest)
m5addamsa                       //
m5addamsb                       //
...
m5xchn                          // Exchanges Unlimited (Barcrest)
m5xena                          // Xena Warrior Princess (Bwb)
m5xfact                         // X Factor (Empire)
m5xfact02                       //
m5xfact04                       //
m5xfact11                       //
m5zigzag                        // Zig Zag
*/



/*
@source:rastersp.cpp
fbcrazy                         // 1997
rotr                            // 1994
*/




Before I kept getting errors:

/usr/bin/time make -j2 | nl
1 GCC 6.3.0 detected
2 Compiling src/mame/drivers/mpu4sw.cpp...
3 Compiling src/mame/drivers/bfm_sc4.cpp...
virtual memory exhausted: Cannot allocate memory


/usr/bin/time make -j2 | nl
1 GCC 6.3.0 detected
2 Compiling src/mame/drivers/bfm_sc4.cpp...
3 Compiling src/mame/drivers/mpu4sw.cpp...

cc1plus: out of memory allocating 64 bytes after a total of 97579008 bytes
make[2]: *** [../../../../linux_gcc/obj/x32/Release/src/mame/drivers/bfm_sc4.o] Error 1
make[1]: *** [bfm] Error 2
4 bfm.make:421: recipe for target '../../../../linux_gcc/obj/x32/Release/src/mame/drivers/bfm_sc4.o' failed
5 Makefile:133: recipe for target 'bfm' failed
make[1]: *** Waiting for unfinished jobs....

Command terminated by signal 13
469.03user 13.58system 12:19.34elapsed 65%CPU (0avgtext+0avgdata 2310700maxresident)k
158600inputs+3104outputs (4343major+3459030minor)pagefaults 0swaps


/usr/bin/time make -j1 | nl
1 GCC 6.3.0 detected
2 Compiling src/mame/drivers/mpu4sw.cpp...
virtual memory exhausted: Cannot allocate memory
make[2]: *** [../../../../linux_gcc/obj/x32/Release/src/mame/drivers/mpu4sw.o] Error 1
3 barcrest.make:472: recipe for target '../../../../linux_gcc/obj/x32/Release/src/mame/drivers/mpu4sw.o' failed
make[1]: *** [barcrest] Error 2
4 Makefile:130: recipe for target 'barcrest' failed
make: *** [linux_x86] Error 2
5 makefile:1252: recipe for target 'linux_x86' failed
Command exited with non-zero status 2
437.82user 11.76system 7:49.94elapsed 95%CPU (0avgtext+0avgdata 2801344maxresident)k
272928inputs+1920outputs (2137major+3818599minor)pagefaults 0swaps


and after I remove barcrest and bfm, no problems running to the finish.
Code
8206	Compiling src/devices/machine/ie15.cpp...
8213	Archiving liboptional.a...
8248	Compiling src/mame/mame.cpp...
8248	Building driver list...
22746 driver(s) found
8250	Compiling generated/version.cpp...
8250	Compiling generated/mame/mame/drivlist.cpp...
8256	Linking mame...
14833.21user 782.10system 2:18:23elapsed 188%CPU (0avgtext+0avgdata 2556296maxresident)k
1416840inputs+4395320outputs (2887major+157584857minor)pagefaults 0swaps
8304	DONE - COMPILE COMPLETE
================================================================================
EXECUTING: cat /proc/meminfo | grep "MemTotal"
MemTotal:        3097188 kB
================================================================================
EXECUTING: lscpu | grep "Model name:"
Model name:            Intel(R) Core(TM)2 CPU          6400  @ 2.13GHz
================================================================================
EXECUTING: lscpu
Architecture:          i686
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    1
Core(s) per socket:    2
Socket(s):             1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 15
Model name:            Intel(R) Core(TM)2 CPU          6400  @ 2.13GHz
Stepping:              2
CPU MHz:               2127.973
BogoMIPS:              4255.94
Virtualization:        VT-x
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts aperfmperf eagerfpu pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm tpr_shadow dtherm

================================================================================
EXECUTING: ls -l ./mame
-rwxr-xr-x 1 golden golden 226877720 Mar 21 16:26 ./mame
================================================================================
EXECUTING: md5sum ./mame
3ad480dbe397219dc37c40b82a97029d  ./mame
================================================================================
EXECUTING: gcc --version | head -n 1
gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
================================================================================
EXECUTING: lsb_release -a
Distributor ID:	Debian
Description:	Debian GNU/Linux 9.4 (stretch)
Release:	9.4
Codename:	stretch
================================================================================
EXECUTING: uname -a
Linux debian93i686 4.9.0-4-686 #1 SMP Debian 4.9.65-3 (2017-12-03) i686 GNU/Linux
================================================================================
EXECUTING: date
Wed Mar 21 16:26:46 PDT 2018
================================================================================



Since gcc 5.4 can compile using around 1.3 GB of max resident memory on ubuntu 16.04 32 bit, and gcc 6.3 takes over 2 GB of memory under debian 9.4, it looks like I should use gcc 5.4 for a 32 bit build.

Re: Compile stats [Re: Golden Child] #112940
03/22/18 03:56 AM
03/22/18 03:56 AM
Joined: Feb 2014
Posts: 146
G
Golden Child Offline OP
Senior Member
Golden Child  Offline OP
Senior Member
G
Joined: Feb 2014
Posts: 146
I also had to modify my lua script to account for the different file name for the mame executable under 32 and 64 bit (i686 = mame, x86_64 = mame64)

Code
-- luatime3.lua
-- This will take the input and print the time in seconds before the line.
-- Once complete it will print info about your system and run mame for a few seconds.
--
-- a nice compile command is something like:
-- make clean; /usr/bin/time make -j2 2>&1 | lua5.3 lua_time3.lua | tee COMPILE_LOG.txt | nl




function popenstr(str) local mypipe=io.popen(str,"r") local mystr=mypipe:read("*all") mypipe:close() return mystr end

arch_table={ x86_64 = "mame64", i686 = "mame" }

for archstr,arch_mamestr in pairs(arch_table) do
  if string.find(popenstr("uname -a"),archstr) then 
      mamebinarystr = arch_mamestr 
--    print(archstr,arch_mamestr) 
  end
end

assert(mamebinarystr,"arch not detected")

donestr = "DONE - COMPILE COMPLETE"

function iif(b,t,f) if b then return t  else return f end end 

l = os.time() while true do j = io.read() k = os.time() print (k-l,iif(j==nil,donestr,j)) if j==nil then break end end 

function print_and_execute(str) print(string.rep("=",80).."\n".."EXECUTING: "..str) os.execute(str) end

print_and_execute("cat /proc/meminfo | grep \"MemTotal\"")
print_and_execute("lscpu | grep \"Model name:\"")
print_and_execute("lscpu")
print_and_execute("lspci -nn | grep -E -i \"Display|VGA\"")
print_and_execute("lspci -nn")
print_and_execute("ls -l ./"..mamebinarystr.."")
print_and_execute("md5sum ./"..mamebinarystr.."")
print_and_execute("gcc --version | head -n 1")
print_and_execute("lsb_release -a")
print_and_execute("uname -a")
print_and_execute("date")
print_and_execute("lsmod | grep \"video\\|drm\"")

mamebase = "/usr/bin/time ./"..mamebinarystr.." -seconds_to_run 7 -sdlvideofps -rompath ../../mameroms 2>&1"

mamegame = "pacman"

print_and_execute(mamebase.." "..mamegame.." -video opengl -verbose")

Re: Compile stats [Re: Golden Child] #113145
04/17/18 09:35 AM
04/17/18 09:35 AM
Joined: Feb 2014
Posts: 146
G
Golden Child Offline OP
Senior Member
Golden Child  Offline OP
Senior Member
G
Joined: Feb 2014
Posts: 146
More silly compiling fun: I thought I'd see how well some of my older systems compile mame. Unfortunately, they don't all have a lot of memory. So why not try mounting the network block device nbd.

I've got an old Athlon 64 3500+ (@ 2.2ghz) system with 512mb which just isn't enough to compile mame without massive swapping. I'm compiling mame 194 under 32 bit debian 9.3 without the barcrest or bfm machines (since they're too big to compile with gcc 6.3)

Using the hard drive swap partition it takes almost 16 hours to compile at 37% cpu.

hard drive swap

46231 Compiling src/mame/mame.cpp...
46238 Building driver list...
46240 22746 driver(s) found
46240 Compiling generated/mame/mame/drivlist.cpp...
46249 Compiling generated/version.cpp...
46249 Linking mame...
57573 19374.11user 2021.97system 15:59:33elapsed 37%CPU (0avgtext+0avgdata 454908maxresident)k
57573 183876608inputs+4450072outputs (8856343major+157465795minor)pagefaults 0swaps
57573 DONE - COMPILE COMPLETE

But if I configure the nbd network block device, I can get that down to 10.5 hours minutes with 55% cpu.

nbd network swap

33103 Building driver list...
33106 22746 driver(s) found
33106 Compiling generated/mame/mame/drivlist.cpp...
33115 Compiling generated/version.cpp...
33116 Linking mame...
37641 19389.47user 1339.42system 10:27:20elapsed 55%CPU (0avgtext+0avgdata 448188maxresident)k
37641 162288192inputs+4445856outputs (9418061major+158097144minor)pagefaults 0swaps
37641 DONE - COMPILE COMPLETE

====================

comparing the link phase

hard drive swap = 57573-46249 = 188 minutes to link
nbd swap = 37641-33116 = 75 minutes to link

======================

comparing luaengine compile:

hard drive swap = 25178-6422 = 312 minutes
nbd swap = 11191-6839 = 72 minutes

=========================

It was a little tricky to get the nbd-client and nbd-server working under ubuntu 17.10.

First I installed nbd-client and nbd-server with
Code
sudo apt install nbd-client nbd-server

then I set up an nbd-server configuration file at /etc/nbd-server/config

Code
[generic]
# If you want to run everything as root rather than the nbd user, you
# may either say "root" in the two following lines, or remove them
# altogether. Do not remove the [generic] section, however.
#       user = nbd
#       group = nbd
        includedir = /etc/nbd-server/conf.d
        allowlist = 1

# What follows are export definitions. You may create as much of them as
# you want, but the section header has to be unique.
[nbdfile]
  exportname = /mnt/a/NBDFILE
  port = 9000
# this doesn't show up on port 9000 like it should


I needed to comment out user=nbd and group=nbd for whatever reason. Also the port number seems to be ignored.
Code
sudo mkdir /mnt/a
sudo mount -t tmpfs tmpfs /mnt/a  -o size=$((1024*350/100))M


This creates a 3.5GB tmpfs ramdisk and mounts on /mnt/a. I would create a 4GB ramdisk but I'm a little ram constrained on this 8GB machine.

Code
dd if=/dev/zero of=/mnt/a/NBDFILE count=$((1024*1024*1024*350/100/512)) status=progress

7340032+0 records in
7340032+0 records out
3758096384 bytes (3.8 GB, 3.5 GiB) copied, 8.87334 s, 424 MB/s



This will create a 3.5GB file for your nbd-client to connect to.

Ok, now launch nbd-server and check to see that it's listening (the default port is 10809)

Code
$ nbd-server
$ netstat -tlpen
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name    
tcp6       0      0 :::10809                :::*                    LISTEN      1000       10764011   29036/nbd-server

You should be able to connect on the local machine with nbd-client using address 127.0.0.1 and port 10809.

If you've got allowlist = 1 in your /etc/nbd-server/config file you can do this to get a listing:

Code
$ sudo nbd-client 127.0.0.1 10809 -l

Negotiation: ..
nbdfile


Make sure that the nbd module is loaded or the nbd-client won't work.

Code
sudo modprobe nbd


Code
$ sudo nbd-client 127.0.0.1 10809 -N nbdfile /dev/nbd0 -s

Negotiation: ..size = 3584MB
bs=1024, sz=3758096384 bytes


So this means connect to 127.0.0.1 port 10809 looking for the name nbdfile and mounting it on /dev/nbd0 and using it as swap.



and to disconnect /dev/nbd0 just use the -d option

Code
$ sudo nbd-client -d /dev/nbd0

disconnect, sock, done



and if you want to see more about the nbd-server config file options try "man 5 nbd-server"

Code
$ man 5 nbd-server

NBD-SERVER(5)                                                                                                       NBD-SERVER(5)

NAME
       /etc/nbd-server/config - configuration file for nbd-server


Once we've got the nbd-client connected, let's format that nbd file with a swap.

Code
$ sudo mkswap /dev/nbd0
Setting up swapspace version 1, size = 3.5 GiB (3758092288 bytes)
no label, UUID=xxxx

and let's add it to our swap devices with a higher priority of 1024.
Code
$ sudo swapon /dev/nbd0 -p 1024


and let's check the status

Code
$ sudo swapon -s

Filename				Type		Size	Used	Priority
/dev/sda5                              	partition	39132156	3984	-1
/dev/nbd0                              	partition	3670012	0	1024


and when you want to disconnect your nbd swap

Code
$ sudo swapoff /dev/nbd0
$ sudo nbd-client -d /dev/nbd0



And a few caveats with Ubuntu 17.10:

Supposedly one should be able to ignore the config file with:

Code
$ nbd-server -C /dev/null 9000 /mnt/a/NBDFILE 


and then connect with:

Code
$ sudo nbd-client 127.0.0.1 9000  /dev/nbd0 -s
Warning: the oldstyle protocol is no longer supported.
This method now uses the newstyle protocol with a default export
Negotiation: ..Error: Read failed: End of file
Exiting.


but this causes a segfault
Code
[22091.691859] nbd-server[4096]: segfault at 0 ip 00007f8d99c64cbe sp 00007fff3bb6d638 error 4 in libc-2.26.so[7f8d99bc4000+1d6000]


One other note, I tried running debian 9.3 off of an external usb hard drive along with a gigabit usb adapter doing nbd-client swap and it crashed my machine. I tried it twice and it crashed twice. It probably overwhelmed the usb bus and after switching to the onboard ethernet it ran fine. I didn't get much more performance with a gigabit adapter, probably due to the slowness of the old Athlon 64 3500+ cpu. I would see peaks of 16MB/sec instead of 12MB/sec, but that didn't make it much faster.

Also when I did my compiles, I was running ubuntu 17.10 64-bit on the nbd-server and debian 9.3 32-bit on the nbd-client.


Last edited by Golden Child; 04/17/18 09:52 AM.
Re: Compile stats [Re: Golden Child] #113169
04/20/18 04:48 PM
04/20/18 04:48 PM
Joined: Feb 2014
Posts: 146
G
Golden Child Offline OP
Senior Member
Golden Child  Offline OP
Senior Member
G
Joined: Feb 2014
Posts: 146
More silly nbd compiling fun - why not put the mame source files on an nbd block device ext2 file system as well.

And also I learned something about reproducible builds, the directory path changes the compiler output.



cat mynbdconfig

Code
[generic]
# If you want to run everything as root rather than the nbd user, you
# may either say "root" in the two following lines, or remove them
# altogether. Do not remove the [generic] section, however.
#	user = nbd
#	group = nbd
	includedir = /etc/nbd-server/conf.d
allowlist = 1

# What follows are export definitions. You may create as much of them as
# you want, but the section header has to be unique.
[nbdfileswap]
  exportname = /mnt/a/NBDFILESWAP

[nbdfileext2]
  exportname = /mnt/a/NBDFILEEXT2


and create a 7GB tmpfs with 3.5 GB each for the swap and the ext2.

Code
$ sudo mount -t tmpfs tmpfs /mnt/a -o size=$((1024*700/100))M
$ dd if=/dev/zero of=/mnt/a/NBDFILESWAP count=$((1024**3*350/100/512)) status=progress
$ dd if=/dev/zero of=/mnt/a/NBDFILEEXT2  count=$((1024**3*350/100/512)) status=progress




I run my nbdserver with
Code
$ nbd-server -C mynbdconfig


and on my other system:


Let's get a listing of the nbd devices provided by the server at 192.168.0.1 port 10809.
Code
$ sudo nbd-client 192.168.0.1 10809 -l


Let's connect to the nbd devices provided in the configuration file:

Code
$ sudo nbd-client 192.168.0.1 10809 /dev/nbd0 -N nbdfileswap -s
$ sudo nbd-client 192.168.0.1 10809 /dev/nbd1 -N nbdfileext2


Now get the swap file setup:

Code
$ sudo mkswap /dev/nbd0
$ sudo swapon  /dev/nbd0 -p 1024


And get the ext2 file system setup (you can make it ext4 if you like with -t ext4)

Code
$ sudo mke2fs /dev/nbd1 -t ext4
$ sudo mount /dev/nbd1 /mnt/c -o loop


Mount the ext2 file system on /mnt/c

Code
$ sudo mkdir /mnt/c/mydir
$ sudo chown myuseraccount /mnt/c/mydir
$ sudo chgrp myuseraccount /mnt/c/mydir


I copy a my directory of files to be compiled to my fresh ext2,.

$ cp -r ~/Downloads/mame194_extract /mnt/c/mydir
$ diff -rq /mnt/c/mydir/mame194_extract ~/Downloads/mame194_extract

and everything should be identical.

and when you're done, it's good to dismount and disconnect everything.

Disconnect the ext2 filesystem:

Code
$ sudo umount /mnt/c
$ sudo nbd-client -d /dev/nbd1


Disconnect the nbd swap

Code
$ sudo swapoff /dev/nbd0
$ sudo nbd-client -d /dev/nbd0


And when you want to close the nbd-server run netstat -tlpen and look for the program name nbd-server and kill that PID number, in this case 30947.

Code
$ netstat -tlpen
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name    
tcp6       0      0 :::10809                :::*                    LISTEN      1000       2854400    30947/nbd-server    
$ kill 30947



I ran my usual compile script, but for some reason, the output mame file wasn't matching what I was expecting. Same compiler, same OS, but using the nbd as the filesystem caused different md5sums.

I ran it multiple times with redoing all the setup, fresh ext2 filesystems, and it seemed like I was getting different results every time.

I asked a question over at https://github.com/NetworkBlockDevice/nbd/issues and I was told that the compile path gets embedded into the binary.

https://reproducible-builds.org/docs/build-path/


my "normal compile directory" would be "cd /mnt/c/Downloads/mame194_extract/mame/"


Code
try 1 : directory named just mame194_extract (no Downloads/ at the head and it's 12 bytes less)

===============================================================================
EXECUTING: ls -l ./mame
-rwxr-xr-x 1 golden golden 226877704 Apr 18 12:54 ./mame
================================================================================
EXECUTING: md5sum ./mame
edf1bf33710bd1d52c9720e69e66de7c  ./mame
================================================================================
EXECUTING: gcc --version | head -n 1
gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
================================================================================
EXECUTING: lsb_release -a
Distributor ID: Debian
Description:    Debian GNU/Linux 9.3 (stretch)
Release:        9.3
Codename:       stretch
================================================================================
EXECUTING: uname -a
Linux debian93silver 4.9.0-4-686 #1 SMP Debian 4.9.65-3 (2017-12-03) i686 GNU/Linux
================================================================================
EXECUTING: date
Wed Apr 18 12:55:10 PDT 2018
================================================================================

try 2  path is Downloads/mame194_extract/mame

================================================================================
EXECUTING: ls -l ./mame
-rwxr-xr-x 1 golden golden 226877716 Apr 19 01:36 ./mame
================================================================================
EXECUTING: md5sum ./mame
1525a7a4cc4d2773031ca2627a66581c  ./mame
================================================================================

try 3  path is just Downloads/mame194_extract  (no mame at the end) note that the filesize is 4 characters less

================================================================================
EXECUTING: ls -l ./mame
-rwxr-xr-x 1 golden golden 226877712 Apr 19 08:36 ./mame
================================================================================
EXECUTING: md5sum ./mame
f47ea376cd6fe09af8845e9fdf82e9ee  ./mame
================================================================================


try 4 path is Downloads/mame194_extract/mame  IDENTICAL MD5SUM to try 2!!!

================================================================================
EXECUTING: ls -l ./mame
-rwxr-xr-x 1 golden golden 226877716 Apr 19 12:41 ./mame
================================================================================
EXECUTING: md5sum ./mame
1525a7a4cc4d2773031ca2627a66581c  ./mame
===============================================================================



My little lua script to add some compiling stats and write a log (now with the all important print working directory pwd!)

Code
-- compiletime.lua
-- This will take the input and print the time in seconds before the line.
-- Once complete it will print info about your system and run mame for a few seconds.
--
-- a nice compile command is something like:
-- make clean; /usr/bin/time make -j2 2>&1 | lua5.3 compiletime.lua | tee COMPILE_LOG.txt | nl




function popenstr(str) local mypipe=io.popen(str,"r") local mystr=mypipe:read("*all") mypipe:close() return mystr end

arch_table={ x86_64 = "mame64", i686 = "mame" }

for archstr,arch_mamestr in pairs(arch_table) do
  if string.find(popenstr("uname -a"),archstr) then 
      mamebinarystr = arch_mamestr 
--    print(archstr,arch_mamestr) 
  end
end

assert(mamebinarystr,"arch not detected")

donestr = "DONE - COMPILE COMPLETE"

function iif(b,t,f) if b then return t  else return f end end 

l = os.time() while true do j = io.read() k = os.time() print (k-l,iif(j==nil,donestr,j)) if j==nil then break end end 

function print_and_execute(str) print(string.rep("=",80).."\n".."EXECUTING: "..str) os.execute(str) end

print_and_execute("cat /proc/meminfo | grep \"MemTotal\"")
print_and_execute("lscpu | grep \"Model name:\"")
print_and_execute("lscpu")
print_and_execute("lspci -nn | grep -E -i \"Display|VGA\"")
print_and_execute("lspci -nn")
print_and_execute("pwd")
print_and_execute("ls -l ./"..mamebinarystr.."")
print_and_execute("md5sum ./"..mamebinarystr.."")
print_and_execute("gcc --version | head -n 1")
print_and_execute("lsb_release -a")
print_and_execute("uname -a")
print_and_execute("date")
print_and_execute("lsmod | grep \"video\\|drm\"")

mamebase = "/usr/bin/time ./"..mamebinarystr.." -seconds_to_run 7 -sdlvideofps -rompath ../../mameroms 2>&1"

mamegame = "pacman"

print_and_execute(mamebase.." "..mamegame.." -video opengl -verbose")






Page 2 of 3 1 2 3

Moderated by  R. Belmont 

Who's Online Now
4 registered members (Edstrom, Dorando, B2K24, 1 invisible), 34 guests, and 3 spiders.
Key: Admin, Global Mod, Mod
Shout Box
Forum Statistics
Forums9
Topics8,564
Posts111,885
Members4,805
Most Online225
May 26th, 2014
Powered by UBB.threads™ PHP Forum Software 7.6.1.1
(Release build 20180111)
Page Time: 0.031s Queries: 15 (0.012s) Memory: 5.8021 MB (Peak: 6.0503 MB) Zlib enabled. Server Time: 2018-08-15 18:38:25 UTC