getting the size of a file on the web and checking its download

hebdemnobad · 27-Oct-2014 16:06:12

colleagues:

the downloadFile function makes the vm download a file, triggering a callback when the download is completed.

how can a coder do the following:

1. find out the size of the file that is being downloaded
2. check to see how much data (in bytes or kilobytes or whatever other metric) of the file has been downloaded.

Here is where I imagine the code to be (see comments)

//code to find out how much in bytes/kilobytes of the file has been downloaded so vm can get percentage of download
fun checkdownloaddprogress (pakfilesize, url, destpath, bytesdownloaded_so_far)= 
if pakfilesize >= bytesdownloaded_so_far then
(


// code to get how many bytes of the pak file have been downloaded so far, "download_progress" (I assume it's an integer)

//create gui for download percentage (I can do this myself, no help needed here.)

 let itof download_progress -> fdownload_progress in
 let itof pakfilesize->fpakfilesize in
 let download_progress/. pakfilesize -> fraction_of_pak_downloaded in
 let ftoi (fraction_of_pak_download *. 100.0) -> ipercentage_downloaded in
 	(
 	_fooS strcatn "percentage of pak file downloaded so far is " :: itoa ipercentage_downloaded :: " %"::nil;
 	set  	bytesdownloaded_so_far = bytesdownloaded_so_far + download_progress;
 	checkdownloaddprogress pakfilesize url destpath bytesdownloaded_so_far;
 	);
 0;
)
else

(
_fooS "environment downloaded!";
0;
);

0;;


//excerpt of player code
//code....
//download predetermined .pak file
let "mainwindow/mainwindow.pak" -> destpath in
  let "http://ifam.net/openspace/spacemanshooter1/dan_no_mohawk/spacemanwithtank.pak" -> urlsrc in	
  (
        
//code to check file size, wich returns file size, I'll call it pakfilesize 



        _cacheActivate;
  	downloadFile urlsrc mkfun3 @loadpakfile  [viewstr loadwidget destpath ];
        checkdownloaddprogress pakfilesize url destpath 0;
  );

 //code

Last edited by hebdemnobad (27-Oct-2014 16:09:26)

iri · 27-Oct-2014 16:17:28

hebdemnobad wrote:

1. find out the size of the file that is being downloaded

The server response header should tell that.

hebdemnobad wrote:

check to see how much data (in bytes or kilobytes or whatever other metric) of the file has been downloaded.

In Scol, you can get this downloaded size from the callback (by using strlen) but i don't know if Arkeon provides a specific high level function in os3d.

hebdemnobad · 27-Oct-2014 16:44:18

iri wrote:

hebdemnobad wrote:
1. find out the size of the file that is being downloaded
The server response header should tell that.
hebdemnobad wrote:
check to see how much data (in bytes or kilobytes or whatever other metric) of the file has been downloaded.
In Scol, you can get this downloaded size from the callback (by using strlen) but i don't know if Arkeon provides a specific high level function in os3d.

thx Iri....I have looked at your tutorial on the INETGetURL function. I see that it can provide a continuous stream of data received from a particular url. Is there a way I can get the header from the url? I've pasted the code from the tutorial below to make it easier for you to put in suggestion(s).

fun CBdownload (inet, datas, received, error)=

	if error == 0 then	// download in progress

	(

		let datas -> [d] in

		mutate datas <- [strcat d received];

		0

	)

	else if error == 1 then	// download finished

	(

		let datas -> [d] in

		_createpack d _getmodifypack "examples/network/request_get.html";

		_fooS "Done !";

		0

	)

	else	// an error occurs

	(

		_fooS strcat "An error occurs : " itoa error;

		0

	);;



fun main ()=

	_showconsole;

	

	INETGetURL _channel "http://www.google.com" 0 @CBdownload [""];

	0;;

iri · 27-Oct-2014 16:57:47

I don't find a _GETcurlRequestHeader reference in os3d packages to obtain these datas, Arkeon might be done by another way.

Otherwise to write your own download callback from the os3d code :

In os3d, this callback is (so, called by downloadFile) in os3dlib / tools.pkg :

fun cbDownloadFile(curlobj, p, data, code)=
  let p -> [str url cbfun] in
  if (code == -1) then
  (
    mutate p <- [(strcat str data) _ _];
    0;
  )
  // download finished
  else if (code == 0) then
  (
    set lHTTP_REQUEST = remove_from_list lHTTP_REQUEST curlobj;   
    exec cbfun with [url str];
  )
  else
  (
    _fooS strcatn ">>>>>>>>> Http download failed : "::url::" whith code : "::(itoa code)::nil;
    set lHTTP_REQUEST = remove_from_list lHTTP_REQUEST curlobj;
    exec cbfun with [url nil];
    0;
  );
  0;;

You could write something like that (not tested) :
I added a local variable downloadedSize. it increments with the recevived datas size. Warning, the callback defined by "cbfun" takes 4 arguments instead of 2. The third is the downloaded size, the fourth is the state of the downloading (0 : finished, -1, is running, other : fatal error). The function is ran for each received part.

You can use another way, a global variable for example.

fun octopus_cbDownloadFile(curlobj, p, data, code)=
  let p -> [str url cbfun downloadedSize] in
  if (code == -1) then
  (
    mutate p <- [(strcat str data) _ _ downloadedSize + (strlen data)];
    exec cbfun with [url str downloadedSize code];
    0;
  )
  // download finished
  else if (code == 0) then
  (
    set lHTTP_REQUEST = remove_from_list lHTTP_REQUEST curlobj;   
    exec cbfun with [url str downloadedSize code];
  )
  else
  (
    _fooS strcatn ">>>>>>>>> Http download failed : "::url::" whith code : "::(itoa code)::nil;
    set lHTTP_REQUEST = remove_from_list lHTTP_REQUEST curlobj;
    exec cbfun with [url nil nil code];
    0;
  );
  0;;

fun octopus_downloadFile(file, cbfun)=
  if strIsUrl file then
  let makeHtmlCookie file -> cookie in
  let _CRcurlRequest _channel file -> objcurl in
  (
    _SETcurlOption objcurl CURLOPT_SSL_VERIFYPEER 0;
        
    if (cookie == nil) then nil else
      _SETcurlOptionS objcurl CURLOPT_COOKIE cookie;
    
    set lHTTP_REQUEST = (_CALLcurlRequest objcurl @cbDownloadFile ["" file cbfun 0])::lHTTP_REQUEST;
    hd lHTTP_REQUEST;
  )
  else
  (
    exec cbfun with [file nil];
    nil;
  );;

iri · 27-Oct-2014 17:01:04

hebdemnobad wrote:

I have looked at your tutorial on the INETGetURL function. I see that it can provide a continuous stream of data received from a particular url. Is there a way I can get the header from the url? I've pasted the code from the tutorial below to make it easier for you to put in suggestion(s).

Yes it is correct, the principle is the same. But here, i use another API than used in OS3D. It is not a good choice to mix both.
See above, my previous post.

hebdemnobad · 27-Oct-2014 18:18:17

....thanks for your code but this will take a few days for me to learn about http requests/responses....

how about this.....in the os3d api there is:

getUrl	(	url 	,
params 	,
cbfun 	 
)		
Download an url using the GET method.

Prototype: fun [S S fun [S S] u0] I

Parameters
S	: the url to download
S	: the url parameters ("login=toto&pass=tata")
fun	[S S] u0 : the callback with url and data
Returns
ObjCURL : the new request

and

getHtmlHeader	(	cont 		)	
Get Html header from an http response.

Prototype: fun [S] S

Parameters
S	: the response
Returns
S : the header

is the response what is triggered in the geturl callback? that way I could extract information from the header.

just a thought.

Last edited by hebdemnobad (27-Oct-2014 18:23:09)

arkeon · 27-Oct-2014 18:40:27

fun cbGetContentSize(curlobj, p, data, code)=
  let p -> [str url cbfun] in
  if (code == -1) then
  (
    mutate p <- [(strcat str data) _ _];
    0;
  )
  // download finished
  else if (code == 0) then
  (
    set lHTTP_REQUEST = remove_from_list lHTTP_REQUEST curlobj;
    let getHtmlHeader str -> [header cont] in
    let getNextToValue header "Content-Length:" -> length in
      exec cbfun with [url atoi length];
  )
  else
  (
    _fooS strcatn ">>>>>>>>> Http get content size failed : "::url::" whith code : "::(itoa code)::nil;
    set lHTTP_REQUEST = remove_from_list lHTTP_REQUEST curlobj;
    exec cbfun with [url nil];
    0;
  );
  0;;


/*!  @ingroup toolsdl
  *  \brief Get content size of an url
  *
  *  <b>Prototype:</b> fun [S fun [S I] u0] I
  *
  *  \param S : the url to download
  *  \param fun [S I] u0 : the callback with url and content length
  *
  *  \return ObjCURL : the new request
  **/
fun getUrlContentLenght(file, cbfun)=
  if strIsUrl file then
  let makeHtmlCookie file -> cookie in
  let _CRcurlRequest _channel file -> objcurl in
  (
    _SETcurlOption objcurl CURLOPT_SSL_VERIFYPEER 0;
    _SETcurlOption objcurl CURLOPT_HEADER 1;
    _SETcurlOption objcurl CURLOPT_NOBODY 1;
    
    if (cookie == nil) then nil else
      _SETcurlOptionS objcurl CURLOPT_COOKIE cookie;
    
    set lHTTP_REQUEST = (_CALLcurlRequest objcurl @cbGetContentSize ["" file cbfun])::lHTTP_REQUEST;
    hd lHTTP_REQUEST;
  )
  else
  (
    exec cbfun with [file nil];
    nil;
  );;

and

fun cbGetContentLength(url, size)=
  _DLGMessageBox _channel nil "curl test" strcat (itoa (size / 1024)) "Ko" 0;  
  0;;


fun main()=	
  _showconsole;
  getUrlContentLenght "http://ifam.net/openspace/spacemanshooter1/dan_no_mohawk/spacemanwithtank.pak" @cbGetContentLength;
  
	0;;

arkeon · 27-Oct-2014 18:41:07

Have been added in the tools for next release

So you should get the url content size first, then download after.
This make the http request without asking for the content "NOBODY" so this is quick.
And read the http header to get the content size.

hebdemnobad · 27-Oct-2014 18:46:01

thx arkeon, I'll try that and report back. and thx iri for offering your help too.

hebdemnobad · 27-Oct-2014 19:01:54

It works!!!!!! thanks!!!!!

hebdemnobad · 27-Oct-2014 19:15:40

so to complete this component (a loading bar for a .pak file)....
can the vm query the size of a .pak file before the downloading is complete (before downloadFile executes its callback)?

arkeon · 27-Oct-2014 19:28:30

first you get the file size with getUrlContentLenght
then you take the code from cbDownloadFile and make your own callback
where you use len of the new incoming data. and compute the percent of this data with the total file length.

arkeon · 27-Oct-2014 20:03:29

I think I'll modify the player to manage the case the pak file is from an url.
So the player will be ready for your needs.

hebdemnobad · 27-Oct-2014 20:27:34

thx arkeon, in this case maybe I can save you some work (although my code, cest fait d'un bricoleur):
this is enough to build a loading bar that will allow a user know how bit a scene he/she is downloading.

//start downloading the file with INET function, add parameters to callback so player can do what it must do
let "dan_no_mohawk/spacemanwithtank.pak" -> destpath in
  let "http://ifam.net/openspace/spacemanshooter1/dan_no_mohawk/spacemanwithtank.pak" -> urlsrc in	
  (
    _cacheActivate;
    	set bytessofar = 0;
        INETGetURL 	_channel "http://ifam.net/openspace/spacemanshooter1/dan_no_mohawk/spacemanwithtank.pak" 0 mkfun5 @CBdownload [urlsrc viewstr loadwidget destpath ] nil;

callback that prints amount of bytes printed continuosly, incerementing data until all data in url loaded:

fun CBdownload (inet, datas, received, error, extra_parameters)=

	let extra_parameters -> [urlsrc viewstr loadwidget destpath ] in
	(
		if error == 0 then	// download in progress

	(

		let datas -> [d] in

		(	mutate datas <- [strcat d received];
			_fooS strcatn  (itoa (strlen received)):: "bytes received. ":: "\n"::nil ;
			set bytessofar = bytessofar + strlen received; 
			_fooS strcatn  (itoa bytessofar):: "bytes received so far. ":: "\n"::nil ;
		);

		0

	)

	else if error == 1 then	// download finished

	(

		let datas -> [d] in
		_storepack d destpath;		
		//_createpack d _getmodifypack "network/downloadepak.pak";
		_fooS strcatn "Done!":: (itoa bytessofar):: " bytes have been downloaded in the file in total"::nil;
		   //unpak / read pak file
    		set sCurrentScenePath = destpath;

   		set loadedProject = crProject destpath nil 1;
    		set loadedProject.PRJ_xmlFile = XMLloadString mkUnpakProject destpath;
    		_fooS strcatn "pak file downloaded and unpacked from: ":: destpath :: " "::nil; 
    		setProjectCbSceneLoaded loadedProject mkfun2 @cbProjectLoaded [viewstr loadwidget];
    		loadProject loadedProject viewstr nil nil nil nil 1; 
    		0

	)

	else	// an error occurs

	(

		_fooS strcat "An error occurs : " itoa error;

		0

	);
	);;

Iri this mixes the core scol api and os3d api, but I don't see any problems here. It all works together.

arkeon · 27-Oct-2014 20:28:46

What's iri mean is you should not mix INET API with Curl API that's all

hebdemnobad · 27-Oct-2014 20:34:47

arkeon wrote:

What's iri mean is you should not mix INET API with Curl API that's all

I see...I don't think that's a problem here as .pak files load only with INET API. I guess I'll try uploading a scene that uses the picture gallery plugit that uses Curl and see what happens.

Last edited by hebdemnobad (27-Oct-2014 20:36:18)

arkeon · 27-Oct-2014 20:36:39

INET api now use Boost::asio since Curl API use CURL.
CURL api if more complete to acces all data types and manage https certificates and cookies

iri · 27-Oct-2014 21:13:45

Like Arkeon, mix two APIs is not a problem, your code works. It is just less easy to maintain the code.

arkeon · 27-Oct-2014 21:15:45

got the project loading, now I'm doing the progress bar.
But I'm wondering if I use HTML interface or just bitmap.

iri · 27-Oct-2014 21:20:50

arkeon wrote:

first you get the file size with getUrlContentLenght
then you take the code from cbDownloadFile and make your own callback
where you use len of the new incoming data. and compute the percent of this data with the total file length.

Ok, but i don't understand why you add this step with getUrlContentLenght ?
With a GET request (also with POST), the header response provides the body size (field "Content-Length"). So, this HEAD request before takes a supplemental time and resource. No ?

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html (14.13)

arkeon · 27-Oct-2014 21:30:15

No the NOBODY flag make the request return only the header.
otherwise you have to download all the file content before searching for the header.

hebdemnobad · 27-Oct-2014 21:32:01

alas I spoke too soon,

hubris

the project was loading because it had been saved to the cache with downloadFile and I hadn't deleted it.
The INET method, while it allows for getting data amounts as they download, for some reason is not saving the file.

I state the url and destpath here, beginning the INET download:

 let "chatwindow1/chatwindow.pak" -> destpath in
  let "http://www.ifam.net/openspace/chatwindow1/chatwindow.pak" -> urlsrc in	
  (
    _cacheActivate;
    	set bytessofar = 0;
        INETGetURL 	_channel "http://www.ifam.net/openspace/chatwindow1/chatwindow.pak" 0 mkfun5 @CBdownload [urlsrc viewstr loadwidget destpath ] nil;
  	//downloadFile urlsrc mkfun3 @loadpakfile  [viewstr loadwidget destpath ];
  	//getUrl urlsrc nil @headersize;
  );

but the trouble is here, the vm creates the folder chatwindow1 in the cache, but but not the file. chatwindow.pak

fun CBdownload (inet, datas, received, error, extra_parameters)=

	let extra_parameters -> [urlsrc viewstr loadwidget destpath ] in
	(
		if error == 0 then	// download in progress

	(

		let datas -> [d] in

		(	mutate datas <- [strcat d received];
			_fooS strcatn  (itoa (strlen received)):: "bytes received. ":: "\n"::nil ;
			set bytessofar = bytessofar + strlen received; 
			_fooS strcatn  (itoa bytessofar):: "bytes received so far. ":: "\n"::nil ;
		);

		0;

	)

	else if error == 1 then	// download finished

	(

		let datas -> [d] in

				
		(
		_createpack d _getmodifypack destpath;
		//_storepack d destpath;

		_fooS strcatn "Done!":: (itoa bytessofar):: " bytes have been downloaded in the file in total"::nil;

		 //unpak / read pak file
	 	_fooS destpath;
    	set sCurrentScenePath = destpath;

   	set loadedProject = crProject destpath nil 1;
    	set loadedProject.PRJ_xmlFile = XMLloadString mkUnpakProject destpath;
    	_fooS strcatn "pak file downloaded and unpacked from: ":: destpath :: " "::nil; 
    	setProjectCbSceneLoaded loadedProject mkfun2 @cbProjectLoaded [viewstr loadwidget];
    	loadProject loadedProject viewstr nil nil nil nil 1; 
    	);
    	0;

	)

	else	// an error occurs

	(

		_fooS strcat "An error occurs : " itoa error;

		0;

	);
	);;

arkeon · 27-Oct-2014 21:34:14

//_storepack d destpath; is commented

iri · 27-Oct-2014 21:39:02

arkeon wrote:

No the NOBODY flag make the request return only the header.
otherwise you have to download all the file content before searching for the header.

hummm ! The header should be provided before the body, the client needs it to adjust its behavior. Have you try to read the header directly with a GET CURL request ?

hebdemnobad · 27-Oct-2014 21:44:01

arkeon wrote:

//_storepack d destpath; is commented

yes that's what I thought, after lots of commenting and uncommenting the code.

however

_createpack d _getmodifypack destpath;

still creates the folder and file in the cache.

and everything works! I changed:

		INETGetURL 	_channel "http://ifam.net/openspace/spacemanshooter1/dan_no_mohawk/spacemanwithtank.pak" 0 mkfun5 @CBdownload [urlsrc viewstr loadwidget destpath ] nil;

to

		INETGetURL 	_channel "http://ifam.net/openspace/spacemanshooter1/dan_no_mohawk/spacemanwithtank.pak" 0 mkfun5 @CBdownload [urlsrc viewstr loadwidget destpath ] [""];

, following Iri's example on redmine.

Everything works now. The file downloads, the vm can query the progress of the download, the vm can get the filesize, the project is created, and loads.

I assume there is something about the INET api that likes those [ ] symbols, as I encountered failures when changing [d] to d in the callback.

As to why this change works, I have no idea. I remain very much a chimpanzee. The should be an emoticon for happy monkey. It would fit me well.

Scolring - Forum

#1 27-Oct-2014 16:06:12

getting the size of a file on the web and checking its download

#2 27-Oct-2014 16:17:28

Re: getting the size of a file on the web and checking its download

#3 27-Oct-2014 16:44:18

Re: getting the size of a file on the web and checking its download

#4 27-Oct-2014 16:57:47

Re: getting the size of a file on the web and checking its download

#5 27-Oct-2014 17:01:04

Re: getting the size of a file on the web and checking its download

#6 27-Oct-2014 18:18:17

Re: getting the size of a file on the web and checking its download

#7 27-Oct-2014 18:40:27

Re: getting the size of a file on the web and checking its download

#8 27-Oct-2014 18:41:07

Re: getting the size of a file on the web and checking its download

#9 27-Oct-2014 18:46:01

Re: getting the size of a file on the web and checking its download

#10 27-Oct-2014 19:01:54

Re: getting the size of a file on the web and checking its download

#11 27-Oct-2014 19:15:40

Re: getting the size of a file on the web and checking its download

#12 27-Oct-2014 19:28:30

Re: getting the size of a file on the web and checking its download

#13 27-Oct-2014 20:03:29

Re: getting the size of a file on the web and checking its download

#14 27-Oct-2014 20:27:34

Re: getting the size of a file on the web and checking its download

#15 27-Oct-2014 20:28:46

Re: getting the size of a file on the web and checking its download

#16 27-Oct-2014 20:34:47

Re: getting the size of a file on the web and checking its download

#17 27-Oct-2014 20:36:39

Re: getting the size of a file on the web and checking its download

#18 27-Oct-2014 21:13:45

Re: getting the size of a file on the web and checking its download

#19 27-Oct-2014 21:15:45

Re: getting the size of a file on the web and checking its download

#20 27-Oct-2014 21:20:50

Re: getting the size of a file on the web and checking its download

#21 27-Oct-2014 21:30:15

Re: getting the size of a file on the web and checking its download

#22 27-Oct-2014 21:32:01

Re: getting the size of a file on the web and checking its download

#23 27-Oct-2014 21:34:14

Re: getting the size of a file on the web and checking its download

#24 27-Oct-2014 21:39:02

Re: getting the size of a file on the web and checking its download

#25 27-Oct-2014 21:44:01

Re: getting the size of a file on the web and checking its download

Board footer